Portrait of Pierre-Luc Bacon

Pierre-Luc Bacon

Core Academic Member

pierre-luc.bacon@mila.quebec

Canada CIFAR AI Chair

Assistant Professor, Université de Montréal, Department of Computer Science and Operations Research

Research Topics

Reinforcement Learning

Biography

Pierre-Luc Bacon is an assistant professor at Université de Montréal in the Department of Computer Science and Operations Research (DIRO). He is also a core academic member of Mila – Quebec Artificial Intelligence Institute and IVADO, and holds a Facebook CIFAR AI Chair. Bacon leads a research group that investigates the challenges posed by the curse of the horizon in reinforcement learning and optimal control.

Current Students

Collaborating researcher - Concordia University

Research Intern - McGill University

Collaborating researcher - ÉTS

Diego Calanzone

PhD - Université de Montréal

Professional Master's - Université de Montréal

Collaborating Alumni - Université de Montréal

Co-supervisor :

Giorgi Gamkrelidze

Master's Research - Polytechnique Montréal

Principal supervisor :

Hanane Dagdougui

Master's Research - Université de Montréal

Adrien Goldszal

Collaborating Alumni - Université de Montréal

PhD - Université de Montréal

PhD - Université de Montréal

Collaborating Alumni

Akash Karthikeyan

PhD - Université de Montréal

Gaspard Lambrechts

Postdoctorate - McGill University

Principal supervisor :

Parker Levesque

Master's Research - Université de Montréal

Principal supervisor :

PhD - Université de Montréal

PhD - Université de Montréal

Master's Research - Université de Montréal

PhD - Université de Montréal

Vamsi Krishna Munjuluri V S

Master's Research - Université de Montréal

PhD - Université de Montréal

Anushree Rankawat

PhD - Université de Montréal

Gabriel Sasseville

PhD - Université de Montréal

Postdoctorate - Université de Montréal

PhD - Université de Montréal

Collaborating Alumni - Polytechnique Montréal

Principal supervisor :

Postdoctorate - Université de Montréal

Principal supervisor :

Justin Veilleux

Master's Research - Université de Montréal

Blog Posts

Diagram of the physics informed AI system

February 26, 2026

Refrigerants are warming up the planet. AI can help.

by

Adrien Goldszal

Pierre-Luc Bacon

Read the article

Observation encoder

July 31, 2024

Neural Differential Equations for Temperature Control in Buildings Under Demand Response Programs

by

Clement Gehring

Mathieu Le Cam

Hanane Dagdougui

Pierre-Luc Bacon

Read the article

Direct Behavior Specification via Constrained Reinforcement Learning

August 31, 2022

Direct Behavior Specification via Constrained Reinforcement Learning

by

Julien Roy

Joshua Romoff

Pierre-Luc Bacon

Chris Pal

Read the article

Publications

Maximum entropy GFlowNets with soft Q-learning

Sobhan Mohammadpour

Emmanuel Bengio

Pierre-Luc Bacon

Generative Flow Networks (GFNs) have emerged as a powerful tool for sampling discrete objects from unnormalized distributions, offering a sc… (see more)alable alternative to Markov Chain Monte Carlo (MCMC) methods. While GFNs draw inspiration from maximum entropy reinforcement learning (RL), the connection between the two has largely been unclear and seemingly applicable only in specific cases. This paper addresses the connection by constructing an appropriate reward function, thereby establishing an exact relationship between GFNs and maximum entropy RL. This construction allows us to introduce maximum entropy GFNs, which, in contrast to GFNs with uniform backward policy, achieve the maximum entropy attainable by GFNs without constraints on the state space.

2023-12-31

AISTATS (published)

proceedings.mlr.press

Double Gumbel Q-Learning

David Yu-tung Hui

Aaron Courville

Pierre-Luc Bacon

2023-09-20

NeurIPS.cc/2023/Conference (spotlight)

Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control

Pierre-Luc Bacon

Bellemare Marc-Emmanuel

Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In th… (see more)is work, we provide a fresh perspective on these behaviors by studying the return landscape: the mapping between a policy and a return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which a single update to the policy parameters leads to a wide range of returns. By taking a distributional view of these returns, we map the landscape, characterizing failure-prone regions of policy space and revealing a hidden dimension of policy quality. We show that the landscape exhibits surprising structure by finding simple paths in parameter space which improve the stability of a policy. To conclude, we develop a distribution-aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of a policy. Taken together, our results provide new insight into the optimization, evaluation, and design of agents.

2023-09-20

NeurIPS.cc/2023/Conference (poster)

When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment

Benjamin Eysenbach

Pierre-Luc Bacon

Reinforcement learning (RL) algorithms face two distinct challenges: learning effective representations of past and present observations, an… (see more)d determining how actions influence future returns. Both challenges involve modeling long-term dependencies. The Transformer architecture has been very successful to solve problems that involve long-term dependencies, including in the RL domain. However, the underlying reason for the strong performance of Transformer-based RL methods remains unclear: is it because they learn effective memory, or because they perform effective credit assignment? After introducing formal definitions of memory length and credit assignment length, we design simple configurable tasks to measure these distinct quantities. Our empirical results reveal that Transformers can enhance the memory capability of RL algorithms, scaling up to tasks that require memorizing observations

2023-09-20

NeurIPS.cc/2023/Conference (oral)

Goal-conditioned GFlowNets for Controllable Multi-Objective Molecular Design

Pierre-Luc Bacon

Christopher Pal

Emmanuel Bengio

In recent years, in-silico molecular design has received much attention from the machine learning community. When designing a new compound f… (see more)or pharmaceutical applications, there are usually multiple properties of such molecules that need to be optimised: binding energy to the target, synthesizability, toxicity, EC50, and so on. While previous approaches have employed a scalarization scheme to turn the multi-objective problem into a preference-conditioned single objective, it has been established that this kind of reduction may produce solutions that tend to slide towards the extreme points of the objective space when presented with a problem that exhibits a concave Pareto front. In this work we experiment with an alternative formulation of goal-conditioned molecular generation to obtain a more controllable conditional model that can uniformly explore solutions along the entire Pareto front.

2023-06-22

ICML.cc/2023/Workshop/DeployableGenerativeAI (published)

Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier

Evgenii Nikishin

Pierre-Luc Bacon

Bellemare Marc-Emmanuel

Aaron Courville

2023-01-31

ICLR.cc/2023/Conference (notable)

Block-State Transformers

Jonathan Pilault

Pierre-Luc Bacon

Christopher Pal

State space models (SSMs) have shown impressive results on tasks that require modeling long-range dependencies and efficiently scale to long… (see more) sequences owing to their subquadratic runtime complexity. Originally designed for continuous signals, SSMs have shown superior performance on a plethora of tasks, in vision and audio; however, SSMs still lag Transformer performance in Language Modeling tasks. In this work, we propose a hybrid layer named Block-State Transformer (BST), that internally combines an SSM sublayer for long-range contextualization, and a Block Transformer sublayer for short-term representation of sequences. We study three different, and completely parallelizable, variants that integrate SSMs and block-wise attention. We show that our model outperforms similar Transformer-based architectures on language modeling perplexity and generalizes to longer sequences. In addition, the Block-State Transformer demonstrates more than tenfold increase in speed at the layer level compared to the Block-Recurrent Transformer when model parallelization is employed.

2022-12-31

arXiv (preprint)

Designing Biological Sequences via Meta-Reinforcement Learning and Bayesian Optimization

Pierre-Luc Bacon

The ability to accelerate the design of biological sequences can have a substantial impact on the progress of the medical field. The problem… (see more) can be framed as a global optimization problem where the objective is an expensive black-box function such that we can query large batches restricted with a limitation of a low number of rounds. Bayesian Optimization is a principled method for tackling this problem. However, the astronomically large state space of biological sequences renders brute-force iterating over all possible sequences infeasible. In this paper, we propose MetaRLBO where we train an autoregressive generative model via Meta-Reinforcement Learning to propose promising sequences for selection via Bayesian Optimization. We pose this problem as that of finding an optimal policy over a distribution of MDPs induced by sampling subsets of the data acquired in the previous rounds. Our in-silico experiments show that meta-learning over such ensembles provides robustness against reward misspecification and achieves competitive results compared to existing strong baselines.

2022-09-12

ArXiv (preprint)

Direct Behavior Specification via Constrained Reinforcement Learning

Pierre-Luc Bacon

Christopher Pal

Christopher Pal

The standard formulation of Reinforcement Learning lacks a practical way of specifying what are admissible and forbidden behaviors. Most oft… (see more)en, practitioners go about the task of behavior specification by manually engineering the reward function, a counter-intuitive process that requires several iterations and is prone to reward hacking by the agent. In this work, we argue that constrained RL, which has almost exclusively been used for safe RL, also has the potential to significantly reduce the amount of work spent for reward specification in applied RL projects. To this end, we propose to specify behavioral preferences in the CMDP framework and to use Lagrangian methods to automatically weigh each of these behavioral constraints. Specifically, we investigate how CMDPs can be adapted to solve goal-based tasks while adhering to several constraints simultaneously. We evaluate this framework on a set of continuous control tasks relevant to the application of Reinforcement Learning for NPC design in video games.

2022-06-27

Proceedings of the 39th International Conference on Machine Learning (published)

proceedings.mlr.press

The Primacy Bias in Deep Reinforcement Learning

Evgenii Nikishin

Pierre-Luc Bacon

Aaron Courville

This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore usefu… (see more)l evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias. Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance.

2022-06-27

Proceedings of the 39th International Conference on Machine Learning (published)

proceedings.mlr.press

Continuous-Time Meta-Learning with Forward Mode Differentiation

Guillaume Lajoie

Pierre-Luc Bacon

Drawing inspiration from gradient-based meta-learning methods with infinitely small gradient steps, we introduce Continuous-Time Meta-Learni… (see more)ng (COMLN), a meta-learning algorithm where adaptation follows the dynamics of a gradient vector field. Specifically, representations of the inputs are meta-learned such that a task-specific linear classifier is obtained as a solution of an ordinary differential equation (ODE). Treating the learning process as an ODE offers the notable advantage that the length of the trajectory is now continuous, as opposed to a fixed and discrete number of gradient steps. As a consequence, we can optimize the amount of adaptation necessary to solve a new task using stochastic gradient descent, in addition to learning the initial conditions as is standard practice in gradient-based meta-learning. Importantly, in order to compute the exact meta-gradients required for the outer-loop updates, we devise an efficient algorithm based on forward mode differentiation, whose memory requirements do not scale with the length of the learning trajectory, thus allowing longer adaptation in constant memory. We provide analytical guarantees for the stability of COMLN, we show empirically its efficiency in terms of runtime and memory usage, and we illustrate its effectiveness on a range of few-shot image classification problems.

2022-01-27

ICLR.cc/2022/Conference (spotlight)

Long-Term Credit Assignment via Model-based Temporal Shortcuts

Pierre-Luc Bacon

2021-12-12

NeurIPS.cc/2021/Workshop/DeepRL (unknown)