Pierre-Luc Bacon

Biography

Pierre-Luc Bacon is an assistant professor at Université de Montréal in the Department of Computer Science and Operations Research (DIRO). He is also a core academic member of Mila – Quebec Artificial Intelligence Institute and IVADO, and holds a Facebook CIFAR AI Chair. Bacon leads a research group that investigates the challenges posed by the curse of the horizon in reinforcement learning and optimal control.

Current Students

Mahdieh Adib

Postdoctorate - Concordia University

mahdieh.adib@mila.quebec

Reza Alvandi

Research Intern - McGill University

reza.alvandi@mila.quebec

Pablo Boitel

Master's Research - École de technologie suprérieure

pablo.boitel@mila.quebec

diego.calanzone@mila.quebec

Diego Calanzone

PhD - Université de Montréal

PhD - Université de Montréal

Co-supervisor :

Emma Frejinger

florence.cloutier@mila.quebec

Kamen Damov

Professional Master's - Université de Montréal

kamen.damov@mila.quebec

giorgi.gamkrelidze@mila.quebec

Giorgi Gamkrelidze

Master's Research - Polytechnique Montréal

Principal supervisor :

Hanane Dagdougui

Arielle Gazzé

Master's Research - Université de Montréal

arielle.gazze@mila.quebec

Niki Howe

PhD - Université de Montréal

Austin Jang

PhD - Université de Montréal

doseok.jang@mila.quebec

akash.karthikeyan@mila.quebec

Akash Karthikeyan

PhD - Université de Montréal

Postdoctorate - McGill University

Principal supervisor :

Aditya Mahajan

gaspard.lambrechts@mila.quebec

Lu Li

PhD - Université de Montréal

Michel Ma

PhD - Université de Montréal

michel.ma@mila.quebec

artiom.matvei@mila.quebec

Artiom Matvei

Master's Research - Université de Montréal

Aneri Muni

PhD - Université de Montréal

aneri.muni@mila.quebec

Vamsi Krishna Munjuluri V S

Master's Research - Université de Montréal

mvkrishna2001@gmail.com

Tianwei Ni

PhD - Université de Montréal

tianwei.ni@mila.quebec

PhD - Université de Montréal

anushree.rankawat@mila.quebec

Gabriel Sasseville

PhD - Université de Montréal

gabriel.sasseville@mila.quebec

Postdoctorate - Université de Montréal

borna.sayedana@mila.quebec

vincent.taboga@mila.quebec

Shu Ai Song

Collaborating researcher - Université de Montréal

shu.ai-song@mila.quebec

Yihao Sun

PhD - Université de Montréal

yihao.sun@mila.quebec

Collaborating Alumni - Polytechnique Montréal

Postdoctorate - Université de Montréal

Principal supervisor :

Yashar Hezaveh

wassim.tenachi@mila.quebec

justin.veilleux@mila.quebec

Justin Veilleux

Master's Research - Université de Montréal

Xin Yan

Master's Research - Université de Montréal

xin.yan@mila.quebec

Blog Posts

Diagram of the physics informed AI system

February 26, 2026

Refrigerants are warming up the planet. AI can help.

Adrien Goldszal

Pierre-Luc Bacon

Read the article

July 31, 2024

Neural Differential Equations for Temperature Control in Buildings Under Demand Response Programs

Vincent Taboga

Clement Gehring

Mathieu Le Cam

Hanane Dagdougui

Pierre-Luc Bacon

Read the article

August 31, 2022

Direct Behavior Specification via Constrained Reinforcement Learning

Julien Roy

Roger Girgis

Joshua Romoff

Pierre-Luc Bacon

Chris Pal

Read the article

Publications

Discovery of Sustainable Refrigerants through Physics-Informed RL Fine-Tuning of Sequence Models

Most refrigerants currently used in air-conditioning systems, such as hydrofluorocarbons, are potent greenhouse gases and are being phased d… (see more)own. Large-scale molecular screening has been applied to the search for alternatives, but in practice only about 300 refrigerants are known, and only a few additional candidates have been suggested without experimental validation. This scarcity of reliable data limits the effectiveness of purely data-driven methods. We present Refgen, a generative pipeline that integrates machine learning with physics-grounded inductive biases. Alongside fine-tuning for valid molecular generation, Refgen incorporates predictive models for critical properties, equations of state, thermochemical polynomials, and full vapor compression cycle simulations. These models enable reinforcement learning fine-tuning under thermodynamic constraints, enforcing consistency and guiding discovery toward molecules that balance efficiency, safety, and environmental impact. By embedding physics into the learning process, Refgen leverages scarce data effectively and enables de novo refrigerant discovery beyond the known set of compounds.

2026-03-14

AI4X @ Neural Information Processing Systems (oral)

Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models

Zhilong Zhang

Haoxiang Ren

Yihao Sun

Yifei Sheng

Haonan Wang

Haoxin Lin

Zhichao Wu

Yang Yu

Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is … (see more)constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across multimodal models and model-based RL, we propose **VLA-MBPO**, a practical world model-based RL framework to tackle these problems in VLA finetuning. Our approach is guided by three key design choices: (i) adapting *unified multimodal models (UMMs)* to VLA settings, leveraging rich multimodal priors to enable world modeling with limited data; (ii) introducing an *interleaved view decoding* mechanism to enforce consistency across views; and (iii) employing *chunk-level branched rollout* to limit rollout horizons and mitigate error compounding during policy optimization. Our theoretical analysis shows a reduction in value gap of VLA-MBPO, and experiments in both simulated and real-world tasks demonstrate that our method effectively improves policy performance and sample efficiency for VLA finetuning.

2026-03-01

World Models @ International Conference on Learning Representations (published)

What Makes Value Learning Efficient in Residual Reinforcement Learning?

Guozheng Ma

Li Li

Haoyu Wang

Zixuan Liu

Dacheng Tao

Residual reinforcement learning (RL) enables stable online refinement of expressive pretrained policies by freezing the base and learning on… (see more)ly bounded corrections. However, value learning in residual RL poses unique challenges that remain poorly understood. In this work, we identify two key bottlenecks: cold start pathology, where the critic lacks knowledge of the value landscape around the base policy, and structural scale mismatch, where the residual contribution is dwarfed by the base action. Through systematic investigation, we uncover the mechanisms underlying these bottlenecks, revealing that simple yet principled solutions suffice: base-policy transitions serve as an essential value anchor for implicit warmup, and critic normalization effectively restores representation sensitivity for discerning value differences. Based on these insights, we propose DAWN (Data-Anchored Warmup and Normalization), a minimal approach targeting efficient value learning in residual RL. By addressing these bottlenecks, DAWN demonstrates substantial efficiency gains across diverse benchmarks, policy architectures, and observation modalities.

2026-02-10

ArXiv (preprint)

arxiv.org

Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons

When training neural networks, dying neurons -- units becoming inactive or saturated -- are traditionally seen as harmful. This paper sheds … (see more)new light on this phenomenon. By exploring the impact of various hyperparameter configurations on dying neurons during training, we gather insights on how to improve upon sparse training approaches to pruning. We introduce Demon Pruning (DemP), a method that controls the proliferation of dead neurons through a combination of noise injection on active units and a one-cycle schedule regularization strategy, dynamically leading to network sparsity. Experiments on CIFAR-10 and ImageNet datasets demonstrate that DemP outperforms existing dense-to-sparse structured pruning methods, achieving better accuracy-sparsity tradeoffs and accelerating training by up to 3.56

2025-12-30

OpenReview (published)

Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning

Roger Creus Castanyer

Johan Obando-Ceron

Scaling deep reinforcement learning networks is challenging and often results in degraded performance, yet the root causes of this failure m… (see more)ode remain poorly understood. Several recent works have proposed mechanisms to address this, but they are often complex and fail to highlight the causes underlying this difficulty. In this work, we conduct a series of empirical analyses which suggest that the combination of non-stationarity with gradient pathologies, due to suboptimal architectural choices, underlie the challenges of scale. We propose a series of direct interventions that stabilize gradient flow, enabling robust performance across a range of network depths and widths. Our interventions are simple to implement and compatible with well-established algorithms, and result in an effective mechanism that enables strong performance even at large scales. We validate our findings on a variety of agents and suites of environments.

2025-12-30

OpenReview (published)

The Three Regimes of Offline-to-Online Reinforcement Learning

Li, Lu

Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online… (see more) interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online-fine tuning that work well in one setting can fail completely in another. We propose a stability--plasticity principle that can explain this inconsistency: we should preserve the knowledge of pretrained policy or offline dataset during online fine-tuning, whichever is better, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 45 of 63 cases. This work provides a principled framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.

2025-12-30

OpenReview (published)

Understanding Behavioral Metric Learning: A Large-Scale Study on Distracting Reinforcement Learning Environments

Ziyan "Ray" Luo

A key approach to state abstraction is approximating behavioral metrics (notably, bisimulation metrics) in the observation space and embeddi… (see more)ng these learned distances in the representation space. While promising for robustness to task-irrelevant noise, as shown in prior work, accurately estimating these metrics remains challenging, requiring various design choices that create gaps between theory and practice. Prior evaluations focus mainly on final returns, leaving the quality of learned metrics and the source of performance gains unclear. To systematically assess how metric learning works in deep reinforcement learning (RL), we evaluate five recent approaches, unified conceptually as isometric embeddings with varying design choices. We benchmark them with baselines across 20 state-based and 14 pixel-based tasks, spanning 370 task configurations with diverse noise settings. Beyond final returns, we introduce the evaluation of a denoising factor to quantify the encoder's ability to filter distractions. To further isolate the effect of metric learning, we propose and evaluate an isolated metric estimation setting, in which the encoder is influenced solely by the metric loss. Finally, we release an open-source, modular codebase to improve reproducibility and support future research on metric learning in deep RL.

2025-12-30

OpenReview (published)

Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning

Guozheng Ma

Lu Li

Zilin Wang

Li Shen

Dacheng Tao

Effectively scaling up deep reinforcement learning models has proven notoriously difficult due to network pathologies during training, motiv… (see more)ating various targeted interventions such as periodic reset and architectural advances such as layer normalization. Instead of pursuing more complex modifications, we show that introducing static network sparsity alone can unlock further scaling potential beyond their dense counterparts with state-of-the-art architectures. This is achieved through simple one-shot random pruning, where a predetermined percentage of network weights are randomly removed once before training. Our analysis reveals that, in contrast to naively scaling up dense DRL networks, such sparse networks achieve both higher parameter efficiency for network expressivity and stronger resistance to optimization challenges like plasticity loss and gradient interference. We further extend our evaluation to visual and streaming RL scenarios, demonstrating the consistent benefits of network sparsity.

2025-10-05

Proceedings of the 42nd International Conference on Machine Learning (published)

proceedings.mlr.press

Planning with Unified Multimodal Models

Yihao Sun

Zhilong Zhang

Yang Yu

With the powerful reasoning capabilities of large language models (LLMs) and vision-language models (VLMs), many recent works have explored … (see more)using them for decision-making. However, most of these approaches rely solely on language-based reasoning, which limits their ability to reason and make informed decisions. Recently, a promising new direction has emerged with unified multimodal models (UMMs), which support both multimodal inputs and outputs. We believe such models have greater potential for decision-making by enabling reasoning through generated visual content. To this end, we propose Uni-Plan, a planning framework built on UMMs. Within this framework, a single model simultaneously serves as the policy, dynamics model, and value function. In addition, to avoid hallucinations in dynamics predictions, we present a novel approach self-discriminated filtering, where the generative model serves as a self-discriminator to filter out invalid dynamics predictions. Experiments on long-horizon planning tasks show that Uni-Plan substantially improves success rates compared to VLM-based methods, while also showing strong data scalability, requiring no expert demonstrations and achieving better performance under the same training-data size. This work lays a foundation for future research in reasoning and decision-making with UMMs.

2025-09-26

ArXiv (preprint)

arxiv.org

State Entropy Regularization for Robust Reinforcement Learning

Yonatan Ashlag

Uri Koren

Mirco Mutti

Esther Derman

Shie Mannor

State entropy regularization has empirically shown better exploration and sample complexity in reinforcement learning (RL). However, its the… (see more)oretical guarantees have not been studied. In this paper, we show that state entropy regularization improves robustness to structured and spatially correlated perturbations. These types of variation are common in transfer learning but often overlooked by standard robust RL methods, which typically focus on small, uncorrelated changes. We provide a comprehensive characterization of these robustness properties, including formal guarantees under reward and transition uncertainty, as well as settings where the method performs poorly. Much of our analysis contrasts state entropy with the widely used policy entropy regularization, highlighting their different benefits. Finally, from a practical standpoint, we illustrate that compared with policy entropy, the robustness advantages of state entropy are more sensitive to the number of rollouts used for policy evaluation.

2025-09-17

NeurIPS.cc/2025/Conference (oral)

Scaling Trends in Language Model Robustness

Nikolaus Howe

Ian McKenzie

Oskar Hollinsworth

Michał Zając

Tom Tseng

Aaron Tucker

Adam Gleave

Increasing model size has unlocked a dazzling array of capabilities in modern language models. At the same time, even frontier models remain… (see more) vulnerable to jailbreaks and prompt injections, despite concerted efforts to make them robust. As both attack and defense gain access to more compute, and as models become larger, what happens to robustness? We argue that to answer this question requires a \emph{scaling} approach, which we employ in an extensive study of language model robustness across several classification tasks, model families, and adversarial attacks. We find that in the absence of explicit safety training, larger models are not consistently more robust; however, scale improves sample efficiency in adversarial training, though it worsens compute efficiency. Further, we find that increasing attack compute smoothly improves attack success rate against both undefended and adversarially trained models. Finally, after exploring robustness transfer across attacks and threat models, we combine attack and defense scaling rates to study the offense-defense balance. We find that while attack scaling outpaces adversarial training across all models studied, larger adversarially trained models might give defense the advantage in the long run. These results underscore the utility of the scaling lens, and provide a paradigm for evaluating future attacks and defenses on frontier models.

2025-07-14

International Conference on Machine Learning (Accept (spotlight poster))

proceedings.mlr.press

Discrete Compositional Generation via General Soft Operators and Robust Reinforcement Learning

Marco Jiralerspong

Esther Derman

Danilo Vucetic

Nikolay Malkin

Bilun Sun

Tianyu Zhang

Gauthier Gidel

A major bottleneck in scientific discovery consists of narrowing an exponentially large set of objects, such as proteins or molecules, to a … (see more)small set of promising candidates with desirable properties. While this process can rely on expert knowledge, recent methods leverage reinforcement learning (RL) guided by a proxy reward function to enable this filtering. By employing various forms of entropy regularization, these methods aim to learn samplers that generate diverse candidates that are highly rated by the proxy function. In this work, we make two main contributions. First, we show that these methods are liable to generate overly diverse, suboptimal candidates in large search spaces. To address this issue, we introduce a novel unified operator that combines several regularized RL operators into a general framework that better targets peakier sampling distributions. Secondly, we offer a novel, robust RL perspective of this filtering process. The regularization can be interpreted as robustness to a compositional form of uncertainty in the proxy function (i.e., the true evaluation of a candidate differs from the proxy's evaluation). Our analysis leads us to a novel, easy-to-use algorithm we name trajectory general mellowmax (TGM): we show it identifies higher quality, diverse candidates than baselines in both synthetic and real-world tasks. Code: https://github.com/marcojira/tgm.

2025-06-19

ArXiv (preprint)