Portrait de Pierre-Luc Bacon

Pierre-Luc Bacon

Membre académique principal

pierre-luc.bacon@mila.quebec

Chaire en IA Canada-CIFAR

Professeur adjoint, Université de Montréal, Département d'informatique et de recherche opérationnelle

Sujets de recherche

Apprentissage par renforcement

Biographie

Pierre-Luc Bacon est professeur agrégé au Département d'informatique et de recherche opérationnelle de l'Université de Montréal. Il est également membre de Mila – Institut québécois d’intelligence artificielle et d’IVADO et titulaire d'une chaire Facebook-CIFAR. Il dirige un groupe de recherche qui travaille sur le défi posé par la malédiction de l'horizon dans l'apprentissage par renforcement et le contrôle optimal.

Étudiants actuels

Collaborateur·rice de recherche - Concordia

mahdieh.adib@mila.quebec

Stagiaire de recherche - McGill

reza.alvandi@mila.quebec

Collaborateur·rice de recherche - ÉTS

pablo.boitel@mila.quebec

Claire Bouillet

Stagiaire de recherche - Polytechnique

claire.bouillet@mila.quebec

Diego Calanzone

Doctorat - UdeM

diego.calanzone@mila.quebec

Florence Cloutier

Doctorat - UdeM

Co-superviseur⋅e :

florence.cloutier@mila.quebec

Maîtrise professionnelle - UdeM

kamen.damov@mila.quebec

Collaborateur·rice alumni - UdeM

Co-superviseur⋅e :

esther.derman@mila.quebec

Giorgi Gamkrelidze

Maîtrise recherche - Polytechnique

Superviseur⋅e principal⋅e :

Hanane Dagdougui

giorgi.gamkrelidze@mila.quebec

Maîtrise recherche - UdeM

arielle.gazze@mila.quebec

Adrien Goldszal

Collaborateur·rice de recherche - UdeM

adrien.goldszal@polytechnique.edu

Doctorat - UdeM

howeniko@mila.quebec

Doctorat - UdeM

doseok.jang@mila.quebec

Akash Karthikeyan

Doctorat - UdeM

akash.karthikeyan@mila.quebec

Gaspard Lambrechts

Postdoctorat - McGill

Superviseur⋅e principal⋅e :

gaspard.lambrechts@mila.quebec

Parker Levesque

Maîtrise recherche - UdeM

Superviseur⋅e principal⋅e :

parker.levesque@mila.quebec

Doctorat - UdeM

lu.li@mila.quebec

Doctorat - UdeM

michel.ma@mila.quebec

Maîtrise recherche - UdeM

artiom.matvei@mila.quebec

Doctorat - UdeM

aneri.muni@mila.quebec

Vamsi Krishna Munjuluri V S

Maîtrise recherche - UdeM

mvkrishna2001@gmail.com

Doctorat - UdeM

tianwei.ni@mila.quebec

Anushree Rankawat

Doctorat - UdeM

anushree.rankawat@mila.quebec

Gabriel Sasseville

Doctorat - UdeM

gabriel.sasseville@mila.quebec

Postdoctorat - UdeM

borna.sayedana@mila.quebec

Doctorat - UdeM

yihao.sun@mila.quebec

Postdoctorat - Polytechnique

vincent.taboga@mila.quebec

Postdoctorat - UdeM

Superviseur⋅e principal⋅e :

wassim.tenachi@mila.quebec

Justin Veilleux

Maîtrise recherche - UdeM

justin.veilleux@mila.quebec

Billets de blogue

Diagram of the physics informed AI system

26 février 2026

Les réfrigérants réchauffent la planète, mais l’IA peut aider.

par

Adrien Goldszal

Pierre-Luc Bacon

Observation encoder

7 août 2024

Équations différentielles neuronales pour la régulation de la température dans les bâtiments dans le cadre de programmes de réponse à la demande

par

Clement Gehring

Mathieu Le Cam

Hanane Dagdougui

Pierre-Luc Bacon

Direct Behavior Specification via Constrained Reinforcement Learning

31 août 2022

Spécification directe du comportement par apprentissage par renforcement sous contrainte

par

Julien Roy

Joshua Romoff

Pierre-Luc Bacon

Chris Pal

Publications

Generative Meta-Models are Adversarial Classifiers

Diego Calanzone

Pierre-Luc Bacon

We study adversarial prompt detection in LLMs through the geometry of intermediate activations. Our hypothesis is that token-level adversari… (voir plus)al attacks induce hidden states that lie outside the activation manifold associated with typical prompts. To test this, we use Generative Latent Priors (GLP), diffusion-based meta-models trained on LLM residual stream activations, as proxies for this manifold (Luo et al., 2026). We evaluate several training-free anomaly scores derived from GLP, namely Guard-GLP, including reconstruction error, diffusion time estimation, and a density-based estimate using Hutchinson trace estimation. Empirically, GLP reconstruction error separates benign prompts from adversarial prompts and provides a competitive classifier without supervised fine-tuning. We also study GLP as a regularizer for activation steering, showing that denoising edited activations can reduce attack success while limiting over-refusal and nonsensical generations. Overall, our results suggest that generative priors over LLM activations provide a useful interface for both adversarial prompt detection and safer activation-level interventions.

2026-08-04

AIW @ Conference on Language Modeling (publié)

Building2Building: A Large Scale Benchmark for Generalizable Real-World Reinforcement Learning

Justin Veilleux

Doseok Jang

Anushree Rankawat

Pierre-Luc Bacon

Reinforcement learning (RL) has achieved strong results in control, yet learned policies remain brittle to changes in dynamics, action space… (voir plus)s, observation spaces, or goals, a critical limitation for real-world deployment. Existing benchmarks offer limited diversity and complexity, making it difficult to rigorously study transfer, multi-task learning, and meta-learning in RL. We introduce Building2Building (B2B), a large-scale suite of realistic Heating, Ventilation, and Air Conditioning (HVAC) control environments built on EnergyPlus, a state-of-the-art building simulator. B2B is fully compatible with the Gymnasium interface and features a parametric building generator, enabling the systematic generation of diverse building configurations with heterogeneous observation and action spaces. Based on this suite, we define benchmark tasks targeting key open challenges in RL, including goal adaptation, dynamics adaptation, action-space shifts, and cross-domain transfer. By providing a large-scale, diverse, and physically grounded testbed with standardized evaluation protocols, B2B enables systematic investigation of generalization and transfer in continuous control. Beyond advancing research on generalization in RL, this new benchmark also carries significant societal implications by enabling improved HVAC control at scale, one of the most energy-intensive systems in buildings.

2026-07-16

arXiv (prépublication)

The Three Regimes of Offline-to-Online Reinforcement Learning

Pierre-Luc Bacon

Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online… (voir plus) interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online-fine tuning that work well in one setting can fail completely in another. We propose a stability--plasticity principle that can explain this inconsistency: we should preserve the knowledge of pretrained policy or offline dataset during online fine-tuning, whichever is better, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 45 of 63 cases. This work provides a principled framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.

2026-06-15

ContinualRL @ Reinforcement Learning Conference (publié)

Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

Johan Obando-Ceron

Pierre-Luc Bacon

Aaron Courville

Pablo Samuel Castro

Scaling reinforcement learning (RL) to diverse multitask settings remains a central challenge. While recent advances in model-based RL achie… (voir plus)ve strong performance, they rely on planning and complex training pipelines, making it unclear which components are essential for scalability. We revisit this question and argue that the primary driver of scalable multitask RL is not model-based control, but \emph{representation learning}. In particular, we show that combining predictive, model-based representations with high-capacity value function approximation is sufficient to achieve strong performance, even without planning. We evaluate a simple model-free algorithm, MR.Q, coupled with auxiliary predictive objectives into a scalable actor-critic architecture. This approach outperforms a recent world-model-based method and a range of deep RL baselines across a diverse suite of multitask continuous control tasks, while significantly reducing computational overhead and improving wall-clock efficiency. We observe consistent improvements with increased model capacity and show through ablations that predictive representation learning is critical for performance.

2026-06-03

arXiv (prépublication)

Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors

Guozheng Ma

Zilin Wang

Pierre-Luc Bacon

Dacheng Tao

Online reinforcement learning (RL) agents increasingly depend on knowledge acquired offline to achieve practical efficiency. Originally stud… (voir plus)ied in offline-to-online RL, this paradigm now spans foundation model post-training and embodied intelligence, with prior types expanding from offline datasets and pre-trained policies to increasingly diverse knowledge sources such as multimodal foundation models and generative world models. Offline priors have become central to how deep RL is developed and deployed. However, this reliance introduces a challenge that the prevailing benchmark-driven paradigm cannot resolve: because prior validity varies across deployments and shifts during training, no single approach to managing it is universally optimal, and benchmark rankings offer limited guidance for real-world deployments. Rather than pursuing universal solutions, we argue that the field should shift to diagnosis-driven tension management, in which deployment-specific evidence guides how the learner relates to its priors throughout training, enabling both flexible and adaptive deployment. We support this position with a framework characterizing how priors reshape online optimization through three functional roles, controlled experiments demonstrating help-or-hurt reversals, cross-domain evidence from foundation model post-training to embodied intelligence, and engagement with five substantive counterarguments.

2026-05-24

DEMO @ International Conference on Machine Learning (poster)

From Static Policies to Adaptive Priors in Offline Reinforcement Learning

Akash Karthikeyan

Pierre-Luc Bacon

Offline reinforcement learning (RL) has traditionally focused on learning policies for direct deployment under conservative objectives, wher… (voir plus)e uncertainty outside the offline dataset is treated pessimistically to ensure robustness. We argue that this formulation becomes incomplete when an offline-trained policy is subsequently updated through online interaction, as increasingly occurs in modern intelligent systems through test-time adaptation and online fine-tuning. This position paper argues that, in such settings, the objective of offline RL should extend beyond immediate deployment and instead prioritize learning *adaptive policy priors*: policies that preserve the capacity to improve during subsequent interaction through memory, exploration, and self-correction. We formalize this perspective as *adaptive offline reinforcement learning* (AORL), distinguish it from offline-to-online RL, and explain why adaptability becomes important under distributional shift, limited dataset coverage, and changing test-time conditions. We further discuss Bayesian offline RL as one principled direction for constructing adaptive policy priors by preserving epistemic uncertainty over plausible environments. Finally, we outline connections, open challenges, and research directions for treating offline RL as preparation for future experience rather than as a static deployment problem.

2026-05-24

DEMO @ International Conference on Machine Learning (poster)

Rotation-Preserving Supervised Fine-Tuning

Pierre-Luc Bacon

Mohammad Hamdaqa

Supervised fine-tuning (SFT) improves in-domain performance but can degrade out-of-domain (OOD) generalization. Prior work suggests that thi… (voir plus)s degradation is related to changes in dominant singular subspaces of pretrained weight matrices. However, directly identifying loss-sensitive directions with Hessian or Fisher information is computationally expensive at LLM scale. In this work, we propose preserving projected rotations in pretrained singular subspaces as an efficient proxy for Fisher-sensitive directions, which we call Rotation-Preserving Supervised Fine-Tuning (RPSFT). RPSFT penalizes changes in the projected top-

2026-05-22

CATS @ International Conference on Machine Learning (poster)

Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity

Pierre-Luc Bacon

Tail-end risk measures such as static conditional value-at-risk (CVaR) are used in safety-critical applications to prevent rare, yet catastr… (voir plus)ophic events. Unlike risk-neutral objectives, the static CVaR of the return depends on entire trajectories without admitting a recursive Bellman decomposition in the underlying Markov decision process. A classical resolution relies on state augmentation with a continuous variable. However, unless restricted to a specialized class of admissible value functions, this formulation induces sparse rewards and degenerate fixed points. In this work, we propose a novel formulation of the static CVaR objective based on augmentation. Our alternative approach leads to a Bellman operator with: (1) dense per-step rewards; (2) contracting properties on the full space of bounded value functions. Building on this theoretical foundation, we develop risk-averse value iteration and model-free Q-learning algorithms that rely on discretized augmented states. We further provide convergence guarantees and approximation error bounds due to discretization. Empirical results demonstrate that our algorithms successfully learn CVaR-sensitive policies and achieve effective performance-safety trade-offs.

2026-04-29

International Conference on Machine Learning (spotlight)

Discovery of Sustainable Refrigerants through Physics-Informed RL Fine-Tuning of Sequence Models

Adrien Goldszal

Diego Calanzone

Pierre-Luc Bacon

Most refrigerants currently used in air-conditioning systems, such as hydrofluorocarbons, are potent greenhouse gases and are being phased d… (voir plus)own. Large-scale molecular screening has been applied to the search for alternatives, but in practice only about 300 refrigerants are known, and only a few additional candidates have been suggested without experimental validation. This scarcity of reliable data limits the effectiveness of purely data-driven methods. We present Refgen, a generative pipeline that integrates machine learning with physics-grounded inductive biases. Alongside fine-tuning for valid molecular generation, Refgen incorporates predictive models for critical properties, equations of state, thermochemical polynomials, and full vapor compression cycle simulations. These models enable reinforcement learning fine-tuning under thermodynamic constraints, enforcing consistency and guiding discovery toward molecules that balance efficiency, safety, and environmental impact. By embedding physics into the learning process, Refgen leverages scarce data effectively and enables de novo refrigerant discovery beyond the known set of compounds.

2026-03-14

AI4X @ Neural Information Processing Systems (présentation orale)

Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models

Zhilong Zhang

Haoxiang Ren

Yifei Sheng

Haonan Wang

Haoxin Lin

Zhichao Wu

Pierre-Luc Bacon

Yang Yu

Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is … (voir plus)constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across multimodal models and model-based RL, we propose **VLA-MBPO**, a practical world model-based RL framework to tackle these problems in VLA finetuning. Our approach is guided by three key design choices: (i) adapting *unified multimodal models (UMMs)* to VLA settings, leveraging rich multimodal priors to enable world modeling with limited data; (ii) introducing an *interleaved view decoding* mechanism to enforce consistency across views; and (iii) employing *chunk-level branched rollout* to limit rollout horizons and mitigate error compounding during policy optimization. Our theoretical analysis shows a reduction in value gap of VLA-MBPO, and experiments in both simulated and real-world tasks demonstrate that our method effectively improves policy performance and sample efficiency for VLA finetuning.

2026-03-01

World Models @ International Conference on Learning Representations (publié)

What Makes Value Learning Efficient in Residual Reinforcement Learning?

Guozheng Ma

Li Li

Haoyu Wang

Zixuan Liu

Pierre-Luc Bacon

Dacheng Tao

Residual reinforcement learning (RL) enables stable online refinement of expressive pretrained policies by freezing the base and learning on… (voir plus)ly bounded corrections. However, value learning in residual RL poses unique challenges that remain poorly understood. In this work, we identify two key bottlenecks: cold start pathology, where the critic lacks knowledge of the value landscape around the base policy, and structural scale mismatch, where the residual contribution is dwarfed by the base action. Through systematic investigation, we uncover the mechanisms underlying these bottlenecks, revealing that simple yet principled solutions suffice: base-policy transitions serve as an essential value anchor for implicit warmup, and critic normalization effectively restores representation sensitivity for discerning value differences. Based on these insights, we propose DAWN (Data-Anchored Warmup and Normalization), a minimal approach targeting efficient value learning in residual RL. By addressing these bottlenecks, DAWN demonstrates substantial efficiency gains across diverse benchmarks, policy architectures, and observation modalities.

2026-02-10

ArXiv (prépublication)

Long-Horizon Model-Based Offline Reinforcement Learning Without Conservatism

Siamak Ravanbakhsh

Pierre-Luc Bacon

2025-12-31

International Conference on Machine Learning (Accept (regular))