Pierre-Luc Bacon

Biography

Pierre-Luc Bacon is an assistant professor at Université de Montréal in the Department of Computer Science and Operations Research (DIRO). He is also a core academic member of Mila – Quebec Artificial Intelligence Institute and IVADO, and holds a Facebook CIFAR AI Chair. Bacon leads a research group that investigates the challenges posed by the curse of the horizon in reinforcement learning and optimal control.

Current Students

Mahdieh Adib

Collaborating researcher - Concordia University

mahdieh.adib@mila.quebec

claire.bouillet@mila.quebec

Reza Alvandi

Research Intern - McGill University

reza.alvandi@mila.quebec

Pablo Boitel

Collaborating researcher - ÉTS

pablo.boitel@mila.quebec

Claire Bouillet

Research Intern - Polytechnique Montréal

Diego Calanzone

PhD - Université de Montréal

diego.calanzone@mila.quebec

Kamen Damov

Professional Master's - Université de Montréal

kamen.damov@mila.quebec

esther.derman@mila.quebec

Esther Derman

Collaborating Alumni - Université de Montréal

Co-supervisor :

Erick Delage

Master's Research - Polytechnique Montréal

Principal supervisor :

Hanane Dagdougui

giorgi.gamkrelidze@mila.quebec

Arielle Gazzé

Master's Research - Université de Montréal

arielle.gazze@mila.quebec

Adrien Goldszal

Collaborating researcher - Université de Montréal

adrien.goldszal@polytechnique.edu

Niki Howe

PhD - Université de Montréal

Austin Jang

PhD - Université de Montréal

doseok.jang@mila.quebec

akash.karthikeyan@mila.quebec

Akash Karthikeyan

PhD - Université de Montréal

Postdoctorate - McGill University

Principal supervisor :

Aditya Mahajan

gaspard.lambrechts@mila.quebec

Master's Research - Université de Montréal

Principal supervisor :

Yashar Hezaveh

parker.levesque@mila.quebec

Lu Li

PhD - Université de Montréal

Michel Ma

PhD - Université de Montréal

michel.ma@mila.quebec

artiom.matvei@mila.quebec

Artiom Matvei

Master's Research - Université de Montréal

Aneri Muni

PhD - Université de Montréal

aneri.muni@mila.quebec

Vamsi Krishna Munjuluri V S

Master's Research - Université de Montréal

mvkrishna2001@gmail.com

Tianwei Ni

PhD - Université de Montréal

tianwei.ni@mila.quebec

PhD - Université de Montréal

anushree.rankawat@mila.quebec

Gabriel Sasseville

PhD - Université de Montréal

gabriel.sasseville@mila.quebec

Postdoctorate - Université de Montréal

borna.sayedana@mila.quebec

vincent.taboga@mila.quebec

Yihao Sun

PhD - Université de Montréal

yihao.sun@mila.quebec

Postdoctorate - Polytechnique Montréal

Postdoctorate - Université de Montréal

Principal supervisor :

Yashar Hezaveh

wassim.tenachi@mila.quebec

justin.veilleux@mila.quebec

Justin Veilleux

Master's Research - Université de Montréal

Blog Posts

Diagram of the physics informed AI system

February 26, 2026

Refrigerants are warming up the planet. AI can help.

Adrien Goldszal

Pierre-Luc Bacon

Read the article

July 31, 2024

Neural Differential Equations for Temperature Control in Buildings Under Demand Response Programs

Vincent Taboga

Clement Gehring

Mathieu Le Cam

Hanane Dagdougui

Pierre-Luc Bacon

Read the article

August 31, 2022

Direct Behavior Specification via Constrained Reinforcement Learning

Julien Roy

Roger Girgis

Joshua Romoff

Pierre-Luc Bacon

Chris Pal

Read the article

Publications

The Barbados 2018 List of Open Issues in Continual Learning

Tom Schaul

Hado van Hasselt

Joseph. Modayil

Martha White

Adam White

Jean Harb

Shibl Mourad

Bellemare Marc-Emmanuel

We want to make progress toward artificial general intelligence, namely general-purpose agents that autonomously learn how to competently ac… (see more)t in complex environments. The purpose of this report is to sketch a research outline, share some of the most important open issues we are facing, and stimulate further discussion in the community. The content is based on some of our discussions during a week-long workshop held in Barbados in February 2018.

2018-11-15

ArXiv (preprint)

Convergent Tree Backup and Retrace with Function Approximation

Off-policy learning is key to scaling up reinforcement learning as it allows to learn about a target policy from the experience generated by… (see more) a different behavior policy. Unfortunately, it has been challenging to combine off-policy learning with function approximation and multi-step bootstrapping in a way that leads to both stable and efficient algorithms. In this work, we show that the \textsc{Tree Backup} and \textsc{Retrace} algorithms are unstable with linear function approximation, both in theory and in practice with specific examples. Based on our analysis, we then derive stable and efficient gradient-based algorithms using a quadratic convex-concave saddle-point formulation. By exploiting the problem structure proper to these algorithms, we are able to provide convergence guarantees and finite-sample bounds. The applicability of our new analysis also goes beyond \textsc{Tree Backup} and \textsc{Retrace} and allows us to provide new convergence rates for the GTD and GTD2 algorithms without having recourse to projections or Polyak averaging.

2018-07-02

Proceedings of the 35th International Conference on Machine Learning (published)

proceedings.mlr.press

Learning with Options that Terminate Off-Policy

Anna Harutyunyan

Peter Vrancx

Ann Nowé

A temporally abstract action, or an option, is specified by a policy and a termination condition: the policy guides option behavior, and the… (see more) termination condition roughly determines its length. Generally, learning with longer options (like learning with multi-step returns) is known to be more efficient. However, if the option set for the task is not ideal, and cannot express the primitive optimal policy exactly, shorter options offer more flexibility and can yield a better solution. Thus, the termination condition puts learning efficiency at odds with solution quality. We propose to resolve this dilemma by decoupling the behavior and target terminations, just like it is done with policies in off-policy learning. To this end, we give a new algorithm, Q(β), that learns the solution with respect to any termination condition, regardless of how the options actually terminate. We derive Q(β) by casting learning with options into a common framework with well-studied multi-step off-policy learning. We validate our algorithm empirically, and show that it holds up to its motivating claims.

2018-04-28

Proceedings of the AAAI Conference on Artificial Intelligence (published)

OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning

Wei-Di Chang

Reinforcement learning has shown promise in learning policies that can solve complex problems. However, manually specifying a good reward fu… (see more)nction can be difficult, especially for intricate tasks. Inverse reinforcement learning offers a useful paradigm to learn the underlying reward function directly from expert demonstrations. Yet in reality, the corpus of demonstrations may contain trajectories arising from a diverse set of underlying reward functions rather than a single one. Thus, in inverse reinforcement learning, it is useful to consider such a decomposition. The options framework in reinforcement learning is specifically designed to decompose policies in a similar light. We therefore extend the options framework and propose a method to simultaneously recover reward options in addition to policy options. We leverage adversarial methods to learn joint reward-policy options using only observed expert states. We show that this approach works well in both simple and complex continuous control tasks and shows significant performance increases in one-shot transfer learning.

2018-04-28

Proceedings of the AAAI Conference on Artificial Intelligence (published)

When Waiting is not an Option: Learning Options with a Deliberation Cost

Jean Harb

Martin Klissarov

Recent work has shown that temporally extended actions (options) can be learned fully end-to-end as opposed to being specified in advance. W… (see more)hile the problem of "how" to learn options is increasingly well understood, the question of "what" good options should be has remained elusive. We formulate our answer to what "good" options should be in the bounded rationality framework (Simon, 1957) through the notion of deliberation cost. We then derive practical gradient-based learning algorithms to implement this objective. Our results in the Arcade Learning Environment (ALE) show increased performance and interpretability.

2018-04-28

Proceedings of the AAAI Conference on Artificial Intelligence (published)

Constructing Temporal Abstractions Autonomously in Reinforcement Learning

2018-02-28

AI Magazine (published)

Learning Robust Options

Daniel J. Mankowitz

Timothy A. Mann

Shie Mannor

Robust reinforcement learning aims to produce policies that have strong guarantees even in the face of environments/transition models whose … (see more)parameters have strong uncertainty. Existing work uses value-based methods and the usual primitive action setting. In this paper, we propose robust methods for learning temporally abstract actions, in the framework of options. We present a Robust Options Policy Iteration (ROPI) algorithm with convergence guarantees, which learns options that are robust to model uncertainty. We utilize ROPI to learn robust options with the Robust Options Deep Q Network (RO-DQN) that solves multiple tasks and mitigates model misspecification due to model uncertainty. We present experimental results which suggest that policy iteration with linear features may have an inherent form of robustness when using coarse feature representations. In addition, we present experimental results which demonstrate that robustness helps policy iteration implemented on top of deep neural networks to generalize over a much broader range of dynamics than non-robust policy iteration.

2018-02-08

ArXiv (preprint)

Learnings Options End-to-End for Continuous Action Tasks

Martin Klissarov

Jean Harb

We present new results on learning temporally extended actions for continuoustasks, using the options framework (Suttonet al.[1999b], Precup… (see more) [2000]). In orderto achieve this goal we work with the option-critic architecture (Baconet al.[2017])using a deliberation cost and train it with proximal policy optimization (Schulmanet al.[2017]) instead of vanilla policy gradient. Results on Mujoco domains arepromising, but lead to interesting questions aboutwhena given option should beused, an issue directly connected to the use of initiation sets.

2017-11-29

ArXiv (preprint)

The Option-Critic Architecture

Jean Harb

Temporal abstraction is key to scaling up learning and planning in reinforcement learning. While planning with temporally extended actions i… (see more)s well understood, creating such abstractions autonomously from data has remained challenging. We tackle this problem in the framework of options [Sutton, Precup & Singh, 1999; Precup, 2000]. We derive policy gradient theorems for options and propose a new option-critic architecture capable of learning both the internal policies and the termination conditions of options, in tandem with the policy over options, and without the need to provide any additional rewards or subgoals. Experimental results in both discrete and continuous environments showcase the flexibility and efficiency of the framework.

2017-02-12

Proceedings of the AAAI Conference on Artificial Intelligence (published)

A Matrix Splitting Perspective on Planning with Options

2016-12-02

ArXiv (preprint)