Publications

Transformers in Reinforcement Learning: A Survey

Pranav Agarwal

Aamer Abdul Rahman

Pierre-Luc St-Charles

Simon J. D. Prince

S Ebrahimi Kahou

Transformers have significantly impacted domains like natural language processing, computer vision, and robotics, where they improve perform… (voir plus)ance compared to other neural networks. This survey explores how transformers are used in reinforcement learning (RL), where they are seen as a promising solution for addressing challenges such as unstable training, credit assignment, lack of interpretability, and partial observability. We begin by providing a brief domain overview of RL, followed by a discussion on the challenges of classical RL algorithms. Next, we delve into the properties of the transformer and its variants and discuss the characteristics that make them well-suited to address the challenges inherent in RL. We examine the application of transformers to various aspects of RL, including representation learning, transition and reward function modeling, and policy optimization. We also discuss recent research that aims to enhance the interpretability and efficiency of transformers in RL, using visualization techniques and efficient training strategies. Often, the transformer architecture must be tailored to the specific needs of a given application. We present a broad overview of how transformers have been adapted for several applications, including robotics, medicine, language modeling, cloud computing, and combinatorial optimization. We conclude by discussing the limitations of using transformers in RL and assess their potential for catalyzing future breakthroughs in this field.

2023-07-11

ArXiv (prépublication)

Data Imputation with an Autoencoder and MAGIC

Devin Eddington

Andres Felipe Duque Correa

Guy Wolf

Kevin R. Moon

Missing data is a common problem in many applications. Imputing missing values is a challenging task, as the imputations need to be accurate… (voir plus) and robust to avoid introducing bias in downstream analysis. In this paper, we propose an ensemble method that combines the strengths of a manifold learning-based imputation method called MAGIC and an autoencoder deep learning model. We call our method Deep MAGIC. Deep MAGIC is trained on a linear combination of the mean squared error of the original data and the mean squared error of the MAGIC-imputed data. Experimental results on three benchmark datasets show that Deep MAGIC outperforms several state-of-the-art imputation methods, demonstrating its effectiveness and robustness in handling large amounts of missing data.

2023-07-09

International Conference on Sampling Theory and Applications (publié)

AI For Global Climate Cooperation 2023 Competition Proceedings

Yoshua Bengio

Prateek Arun Gupta

Li Li

Soham Rajesh Phade

Sunil Srinivasa

Andrew Robert Williams

Tianyu Zhang

Yang Zhang

Stephan Zheng

The international community must collaborate to mitigate climate change and sustain economic growth. However, collaboration is hard to achie… (voir plus)ve, partly because no global authority can ensure compliance with international climate agreements. Combining AI with climate-economic simulations offers a promising solution to design international frameworks, including negotiation protocols and climate agreements, that promote and incentivize collaboration. In addition, these frameworks should also have policy goals fulfillment, and sustained commitment, taking into account climate-economic dynamics and strategic behaviors. These challenges require an interdisciplinary approach across machine learning, economics, climate science, law, policy, ethics, and other fields. Towards this objective, we organized AI for Global Climate Cooperation, a Mila competition in which teams submitted proposals and analyses of international frameworks, based on (modifications of) RICE-N, an AI-driven integrated assessment model (IAM). In particular, RICE-N supports modeling regional decision-making using AI agents. Furthermore, the IAM then models the climate-economic impact of those decisions into the future. Whereas the first track focused only on performance metrics, the proposals submitted to the second track were evaluated both quantitatively and qualitatively. The quantitative evaluation focused on a combination of (i) the degree of mitigation of global temperature rise and (ii) the increase in economic productivity. On the other hand, an interdisciplinary panel of human experts in law, policy, sociology, economics and environmental science, evaluated the solutions qualitatively. In particular, the panel considered the effectiveness, simplicity, feasibility, ethics, and notions of climate justice of the protocols. In the third track, the participants were asked to critique and improve RICE-N.

2023-07-09

ArXiv (prépublication)

Graph Fourier MMD for Signals on Graphs

Samuel Leone

Aarthi Venkat

While numerous methods have been proposed for computing distances between probability distributions in Euclidean space, relatively little at… (voir plus)tention has been given to computing such distances for distributions on graphs. However, there has been a marked increase in data that either lies on graph (such as protein interaction networks) or can be modeled as a graph (single cell data), particularly in the biomedical sciences. Thus, it becomes important to find ways to compare signals defined on such graphs. Here, we propose Graph Fourier MMD (GFMMD), a novel distance between distributions and signals on graphs. GFMMD is defined via an optimal witness function that is both smooth on the graph and maximizes difference in expectation between the pair of distributions on the graph. We find an analytical solution to this optimization problem as well as an embedding of distributions that results from this method. We also prove several properties of this method including scale invariance and applicability to disconnected graphs. We showcase it on graph benchmark datasets as well on single cell RNA-sequencing data analysis. In the latter, we use the GFMMD-based gene embeddings to find meaningful gene clusters. We also propose a novel type of score for gene selection called "gene localization score" which helps select genes for cellular state space characterization.

2023-07-09

2023 International Conference on Sampling Theory and Applications (SampTA) (publié)

International Institutions for Advanced AI

Lewis Ho

Joslyn N. Barnhart

Robert Trager

Yoshua Bengio

Miles Brundage

Allison Sovey Carnegie

Rumman Chowdhury

Allan Dafoe

Gillian K. Hadfield

Margaret Levi

D. Snidal

2023-07-09

ArXiv (prépublication)

Manifold Alignment with Label Information

Andres F. Duque Correa

Myriam Lizotte

Guy Wolf

Kevin R. Moon

Multi-domain data is becoming increasingly common and presents both challenges and opportunities in the data science community. The integrat… (voir plus)ion of distinct data-views can be used for exploratory data analysis, and benefit downstream analysis including machine learning related tasks. With this in mind, we present a novel manifold alignment method called MALI (Manifold alignment with label information) that learns a correspondence between two distinct domains. MALI belongs to a middle ground between the more commonly addressed semi-supervised manifold alignment, where some known correspondences between the two domains are assumed to be known beforehand, and the purely unsupervised case, where no information linking both domains is available. To do this, MALI learns the manifold structure in both domains via a diffusion process and then leverages discrete class labels to guide the alignment. MALI recovers a pairing and a common representation that reveals related samples in both domains. We show that MALI outperforms the current state-of-the-art manifold alignment methods across multiple datasets.

2023-07-09

2023 International Conference on Sampling Theory and Applications (SampTA) (publié)

Robust and Versatile Bipedal Jumping Control through Reinforcement Learning

Zhongyu Li

Xue Bin Peng

Pieter Abbeel

Sergey Levine

Glen Berseth

Koushil Sreenath

2023-07-09

Robotics: Science and Systems XIX (publié)

Overcoming the Technical Challenges of Coordinating Distributed Load Resources at Scale

Johanna Mathieu

Ian Hiskens

Ioannis Marios Granitsas

Oluwagbemileke Oyefeso

Gregory Ledva

Sebastian Nugroho

Salman Nazir

Scott Hinson

Suzanne Russo

Steve Mock

Rachel Jenkins

Jill Harlow

Grant Fisher

Drew Geller

Duncan Callaway

Phillippe Phanivong

Antoine Lesage-Landry

2023-07-07

(publié)

Adjusting Machine Learning Decisions for Equal Opportunity and Counterfactual Fairness

Yixin Wang

Dhanya Sridhar

David Blei

Machine learning ( ml ) methods have the potential to automate high-stakes decisions, such as bail admissions or credit lending, by analyzin… (voir plus)g and learning from historical data. But these algorithmic decisions may be unfair: in learning from historical data, they may replicate discriminatory practices from the past. In this paper, we propose two algorithms that adjust ﬁtted ML predictors to produce decisions that are fair. Our methods provide post-hoc adjustments to the predictors, without requiring that they be retrained. We consider a causal model of the ML decisions, deﬁne fairness through counterfactual decisions within the model, and then form algorithmic decisions that capture the historical data as well as possible, but are provably fair. In particular, we consider two deﬁnitions of fairness. The ﬁrst is “equal counterfactual opportunity,” where the counterfactual distribution of the decision is the same regardless of the protected attribute; the second is counterfactual fairness. We evaluate the algorithms, and the trade-o � between accuracy and fairness, on datasets about admissions, income, credit, and recidivism.

2023-07-06

TMLR (accepté)

Capacity Planning in Stable Matching: An Application to School Choice

Federico Bobbio

Margarida Carvalho

Andrea Lodi

Ignacio Rios

Alfredo Torrico

Centralized mechanisms are becoming the standard approach to solve several assignment problems. Examples include the allocation of students … (voir plus)to schools (school choice), high-school graduates to colleges, residents to hospitals and refugees to cities. In most of these markets, a desirable property of the assignment is stability, which guarantees that no pair of agents has incentive to circumvent the matching. Using school choice as our matching market application, we introduce the problem of jointly allocating a school capacity expansion and finding the best stable matching for the students in the expanded market. We analyze theoretically the problem, focusing on the trade-off behind the multiplicity of student-optimal assignments, and the problem complexity. Since the theoretical intractability of the problem precludes the adaptation of classical approaches to solve it efficiently, we generalize existent mathematical programming formulations of stability constraints to our setting. These generalizations result in integer quadratically-constrained programs, which are computationally hard to solve. In addition, we propose a novel mixed-integer linear programming formulation that is exponentially-large on the problem size. We show that the stability constraints can be separated in linear time, leading to an effective cutting-plane method. We evaluate the performance of our approaches in a detailed computational study, and we find that our cutting-plane method outperforms mixed-integer programming solvers applied to existent formulations extended to our problem setting. We also propose two heuristics that are effective for large instances of the problem. Finally, we use the Chilean school choice system data to demonstrate the impact of capacity planning under stability conditions. Our results show that each additional school seat can benefit multiple students. On the one hand, we can focus on access by prioritizing extra seats that benefit previously unassigned students; on the other hand, we can focus on merit by allocating extra seats that benefit several students via chains of improvement. These insights empower the decision-maker in tuning the matching algorithm to provide a fair application-oriented solution.

2023-07-06

Proceedings of the 24th ACM Conference on Economics and Computation (publié)

Generative Adversarial Nets

Moez Krichen

Ian J. Goodfellow

Mehdi Mirza

Generative Adversarial Networks (GANs) are very popular frameworks for generating high-quality data, and are immensely used in both the acad… (voir plus)emia and industry in many domains. Arguably, their most substantial impact has been in the area of computer vision, where they achieve state-of-the-art image generation. This chapter gives an introduction to GANs, by discussing their principle mechanism and presenting some of their inherent problems during training and evaluation. We focus on these three issues: (1) mode collapse, (2) vanishing gradients, and (3) generation of low-quality images. We then list some architecture-variant and loss-variant GANs that remedy the above challenges. Lastly, we present two utilization examples of GANs for real-world applications: Data augmentation and face images generation.

2023-07-05

2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT) (publié)