Pierre-Luc Bacon

Biographie

Pierre-Luc Bacon est professeur agrégé au Département d'informatique et de recherche opérationnelle de l'Université de Montréal. Il est également membre de Mila – Institut québécois d’intelligence artificielle et d’IVADO et titulaire d'une chaire Facebook-CIFAR. Il dirige un groupe de recherche qui travaille sur le défi posé par la malédiction de l'horizon dans l'apprentissage par renforcement et le contrôle optimal.

Étudiants actuels

Mahdieh Adib

Collaborateur·rice de recherche - Concordia

mahdieh.adib@mila.quebec

claire.bouillet@mila.quebec

Reza Alvandi

Stagiaire de recherche - McGill

reza.alvandi@mila.quebec

Pablo Boitel

Collaborateur·rice de recherche - ÉTS

pablo.boitel@mila.quebec

Claire Bouillet

Stagiaire de recherche - Polytechnique

Diego Calanzone

Doctorat - UdeM

diego.calanzone@mila.quebec

Kamen Damov

Maîtrise professionnelle - UdeM

kamen.damov@mila.quebec

esther.derman@mila.quebec

Esther Derman

Collaborateur·rice alumni - UdeM

Co-superviseur⋅e :

Erick Delage

Maîtrise recherche - Polytechnique

Superviseur⋅e principal⋅e :

Hanane Dagdougui

giorgi.gamkrelidze@mila.quebec

Arielle Gazzé

Maîtrise recherche - UdeM

arielle.gazze@mila.quebec

Adrien Goldszal

Collaborateur·rice de recherche - UdeM

adrien.goldszal@polytechnique.edu

Niki Howe

Doctorat - UdeM

Austin Jang

Doctorat - UdeM

doseok.jang@mila.quebec

akash.karthikeyan@mila.quebec

Akash Karthikeyan

Doctorat - UdeM

Postdoctorat - McGill

Superviseur⋅e principal⋅e :

Aditya Mahajan

gaspard.lambrechts@mila.quebec

Maîtrise recherche - UdeM

Superviseur⋅e principal⋅e :

Yashar Hezaveh

parker.levesque@mila.quebec

Lu Li

Doctorat - UdeM

Michel Ma

Doctorat - UdeM

michel.ma@mila.quebec

artiom.matvei@mila.quebec

Artiom Matvei

Maîtrise recherche - UdeM

Aneri Muni

Doctorat - UdeM

aneri.muni@mila.quebec

Vamsi Krishna Munjuluri V S

Maîtrise recherche - UdeM

mvkrishna2001@gmail.com

Tianwei Ni

Doctorat - UdeM

tianwei.ni@mila.quebec

Doctorat - UdeM

anushree.rankawat@mila.quebec

Gabriel Sasseville

Doctorat - UdeM

gabriel.sasseville@mila.quebec

Postdoctorat - UdeM

borna.sayedana@mila.quebec

vincent.taboga@mila.quebec

Yihao Sun

Doctorat - UdeM

yihao.sun@mila.quebec

Postdoctorat - Polytechnique

Postdoctorat - UdeM

Superviseur⋅e principal⋅e :

Yashar Hezaveh

wassim.tenachi@mila.quebec

justin.veilleux@mila.quebec

Justin Veilleux

Maîtrise recherche - UdeM

Billets de blogue

Diagram of the physics informed AI system

26 février 2026

Les réfrigérants réchauffent la planète, mais l’IA peut aider.

par

Adrien Goldszal

Pierre-Luc Bacon

Lire l'article

7 août 2024

Équations différentielles neuronales pour la régulation de la température dans les bâtiments dans le cadre de programmes de réponse à la demande

par

Vincent Taboga

Clement Gehring

Mathieu Le Cam

Hanane Dagdougui

Pierre-Luc Bacon

Lire l'article

Direct Behavior Specification via Constrained Reinforcement Learning

31 août 2022

Spécification directe du comportement par apprentissage par renforcement sous contrainte

par

Julien Roy

Roger Girgis

Joshua Romoff

Pierre-Luc Bacon

Chris Pal

Lire l'article

Publications

Myriad: a real-world testbed to bridge trajectory optimization and deep learning

Nikolaus H. R. Howe

Simon Dufort-Labbé

Nitarshan Rajkumar

We present Myriad, a testbed written in JAX for learning and planning in real-world continuous environments. The primary contributions of My… (voir plus)riad are threefold. First, Myriad provides machine learning practitioners access to trajectory optimization techniques for application within a typical automatic differentiation workflow. Second, Myriad presents many real-world optimal control problems, ranging from biology to medicine to engineering, for use by the machine learning community. Formulated in continuous space and time, these environments retain some of the complexity of real-world systems often abstracted away by standard benchmarks. As such, Myriad strives to serve as a stepping stone towards application of modern machine learning techniques for impactful real-world tasks. Finally, we use the Myriad repository to showcase a novel approach for learning and control tasks. Trained in a fully end-to-end fashion, our model leverages an implicit planning module over neural ordinary differential equations, enabling simultaneous learning and planning with complex environment dynamics.

2022-11-28

Neural Information Processing Systems (Accept)

Designing Biological Sequences via Meta-Reinforcement Learning and Bayesian Optimization

The ability to accelerate the design of biological sequences can have a substantial impact on the progress of the medical field. The problem… (voir plus) can be framed as a global optimization problem where the objective is an expensive black-box function such that we can query large batches restricted with a limitation of a low number of rounds. Bayesian Optimization is a principled method for tackling this problem. However, the astronomically large state space of biological sequences renders brute-force iterating over all possible sequences infeasible. In this paper, we propose MetaRLBO where we train an autoregressive generative model via Meta-Reinforcement Learning to propose promising sequences for selection via Bayesian Optimization. We pose this problem as that of finding an optimal policy over a distribution of MDPs induced by sampling subsets of the data acquired in the previous rounds. Our in-silico experiments show that meta-learning over such ensembles provides robustness against reward misspecification and achieves competitive results compared to existing strong baselines.

2022-09-12

ArXiv (prépublication)

Direct Behavior Specification via Constrained Reinforcement Learning

Christopher Pal

The standard formulation of Reinforcement Learning lacks a practical way of specifying what are admissible and forbidden behaviors. Most oft… (voir plus)en, practitioners go about the task of behavior specification by manually engineering the reward function, a counter-intuitive process that requires several iterations and is prone to reward hacking by the agent. In this work, we argue that constrained RL, which has almost exclusively been used for safe RL, also has the potential to significantly reduce the amount of work spent for reward specification in applied RL projects. To this end, we propose to specify behavioral preferences in the CMDP framework and to use Lagrangian methods to automatically weigh each of these behavioral constraints. Specifically, we investigate how CMDPs can be adapted to solve goal-based tasks while adhering to several constraints simultaneously. We evaluate this framework on a set of continuous control tasks relevant to the application of Reinforcement Learning for NPC design in video games.

2022-07-18

International Conference on Machine Learning (Accept for Short Presentation)

proceedings.mlr.press

The Primacy Bias in Deep Reinforcement Learning

This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore usefu… (voir plus)l evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias. Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance.

2022-06-27

Proceedings of the 39th International Conference on Machine Learning (publié)

proceedings.mlr.press

Continuous-Time Meta-Learning with Forward Mode Differentiation

Drawing inspiration from gradient-based meta-learning methods with infinitely small gradient steps, we introduce Continuous-Time Meta-Learni… (voir plus)ng (COMLN), a meta-learning algorithm where adaptation follows the dynamics of a gradient vector field. Specifically, representations of the inputs are meta-learned such that a task-specific linear classifier is obtained as a solution of an ordinary differential equation (ODE). Treating the learning process as an ODE offers the notable advantage that the length of the trajectory is now continuous, as opposed to a fixed and discrete number of gradient steps. As a consequence, we can optimize the amount of adaptation necessary to solve a new task using stochastic gradient descent, in addition to learning the initial conditions as is standard practice in gradient-based meta-learning. Importantly, in order to compute the exact meta-gradients required for the outer-loop updates, we devise an efficient algorithm based on forward mode differentiation, whose memory requirements do not scale with the length of the learning trajectory, thus allowing longer adaptation in constant memory. We provide analytical guarantees for the stability of COMLN, we show empirically its efficiency in terms of runtime and memory usage, and we illustrate its effectiveness on a range of few-shot image classification problems.

2022-01-27

ICLR.cc/2022/Conference (spotlight)

Long-Term Credit Assignment via Model-based Temporal Shortcuts

2021-12-12

NeurIPS.cc/2021/Workshop/DeepRL (accepté)

Neural Algorithmic Reasoners are Implicit Planners

Andreea Deac

Petar Veličković

Ognjen Milinković

Jian Tang

Mladen Nikolić

Implicit planning has emerged as an elegant technique for combining learned models of the world with end-to-end model-free reinforcement lea… (voir plus)rning. We study the class of implicit planners inspired by value iteration, an algorithm that is guaranteed to yield perfect policies in fully-specified tabular environments. We find that prior approaches either assume that the environment is provided in such a tabular form -- which is highly restrictive -- or infer "local neighbourhoods" of states to run value iteration over -- for which we discover an algorithmic bottleneck effect. This effect is caused by explicitly running the planning algorithm based on scalar predictions in every state, which can be harmful to data efficiency if such scalars are improperly predicted. We propose eXecuted Latent Value Iteration Networks (XLVINs), which alleviate the above limitations. Our method performs all planning computations in a high-dimensional latent space, breaking the algorithmic bottleneck. It maintains alignment with value iteration by carefully leveraging neural graph-algorithmic reasoning and contrastive self-supervised learning. Across eight low-data settings -- including classical control, navigation and Atari -- XLVINs provide significant improvements to data efficiency against value iteration-based implicit planners, as well as relevant model-free baselines. Lastly, we empirically verify that XLVINs can closely align with value iteration.

2021-12-06

Conference on Neural Information Processing Systems (Accept (Spotlight))

TDprop: Does Adaptive Optimization With Jacobi Preconditioning Help Temporal Difference Learning?

We investigate whether Jacobi preconditioning, accounting for the bootstrap term in temporal difference (TD) learning, can help boost perfor… (voir plus)mance of adaptive optimizers. Our method, TDprop, computes a per-parameter learning rate based on the diagonal preconditioning of the TD update rule. We show how this can be used in both n-step returns and TD(λ). Our theoretical findings demonstrate that including this additional preconditioning information is comparable to normal semi-gradient TD if the optimal learning rate is found for both via a hyperparameter search. This matches our experimental results. In Deep RL experiments using Expected SARSA, TDprop meets or exceeds the performance of Adam in all tested games under near-optimal learning rates, but a well-tuned SGD can yield similar performance in most settings. Our findings suggest that Jacobi preconditioning may improve upon Adam in Deep RL, but despite incorporating additional information from the TD bootstrap term, may not always be better than SGD. Moreover, they suggest that more theoretical investigations are needed to understand adaptive optimizers under optimal hyperparameter regimes in TD learning: simpler methods may, surprisingly, be theoretically comparable after a hyperparameter search.

2021-05-02

Adaptive Agents and Multi-Agent Systems (publié)

Options of Interest: Temporal Abstraction with Interest Functions

Khimya Khetarpal

Martin Klissarov

Maxime Chevalier-Boisvert

Doina Precup

Temporal abstraction refers to the ability of an agent to use behaviours of controllers which act for a limited, variable amount of time. Th… (voir plus)e options framework describes such behaviours as consisting of a subset of states in which they can initiate, an internal policy and a stochastic termination condition. However, much of the subsequent work on option discovery has ignored the initiation set, because of difficulty in learning it from data. We provide a generalization of initiation sets suitable for general function approximation, by defining an interest function associated with an option. We derive a gradient-based learning algorithm for interest functions, leading to a new interest-option-critic architecture. We investigate how interest functions can be leveraged to learn interpretable and reusable temporal abstractions. We demonstrate the efficacy of the proposed approach through quantitative and qualitative results, in both discrete and continuous environments.

2020-04-02

Proceedings of the AAAI Conference on Artificial Intelligence (publié)

Policy Evaluation Networks

Jean Harb

Tom Schaul

Doina Precup

Many reinforcement learning algorithms use value functions to guide the search for better policies. These methods estimate the value of a si… (voir plus)ngle policy while generalizing across many states. The core idea of this paper is to flip this convention and estimate the value of many policies, for a single set of states. This approach opens up the possibility of performing direct gradient ascent in policy space without seeing any new data. The main challenge for this approach is finding a way to represent complex policies that facilitates learning and generalization. To address this problem, we introduce a scalable, differentiable fingerprinting mechanism that retains essential policy information in a concise embedding. Our empirical results demonstrate that combining these three elements (learned Policy Evaluation Network, policy fingerprints, gradient ascent) can produce policies that outperform those that generated the training data, in zero-shot manner.

2020-02-25

ArXiv (prépublication)

TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?

2019-12-31

arXiv (prépublication)

Entropy Regularization with Discounted Future State Distribution in Policy Gradient Methods

The policy gradient theorem is defined based on an objective with respect to the initial distribution over states. In the discounted case, t… (voir plus)his results in policies that are optimal for one distribution over initial states, but may not be uniformly optimal for others, no matter where the agent starts from. Furthermore, to obtain unbiased gradient estimates, the starting point of the policy gradient estimator requires sampling states from a normalized discounted weighting of states. However, the difficulty of estimating the normalized discounted weighting of states, or the stationary state distribution, is quite well-known. Additionally, the large sample complexity of policy gradient methods is often attributed to insufficient exploration, and to remedy this, it is often assumed that the restart distribution provides sufficient exploration in these algorithms. In this work, we propose exploration in policy gradient methods based on maximizing entropy of the discounted future state distribution. The key contribution of our work includes providing a practically feasible algorithm to estimate the normalized discounted weighting of states, i.e, the \textit{discounted future state distribution}. We propose that exploration can be achieved by entropy regularization with the discounted state distribution in policy gradients, where a metric for maximal coverage of the state space can be based on the entropy of the induced state distribution. The proposed approach can be considered as a three time-scale algorithm and under some mild technical conditions, we prove its convergence to a locally optimal policy. Experimentally, we demonstrate usefulness of regularization with the discounted future state distribution in terms of increased state space coverage and faster learning on a range of complex tasks.

2019-12-10

ArXiv (prépublication)