Doina Precup

busra-tugce.gurbuz@mila.quebec

Jesse Farebrother

Doctorat - McGill

Superviseur⋅e principal⋅e :

Marc Gendron-Bellemare

Doctorat - McGill

Superviseur⋅e principal⋅e :

Eilif B. Muller

Mohammad Sami Nur Islam Islam

Jonathan Hu

Maîtrise recherche - McGill

jonathan.hu@mila.quebec

Howard Huang

Doctorat - McGill

huanghow@mila.quebec

Maîtrise recherche - McGill

mohammad-sami-nur.islam@mila.quebec

Hangzhan Jin

Doctorat - Polytechnique

jinhangzhan@gmail.com

flemming.kondrup@mila.quebec

Flemming Kondrup

Postdoctorat - McGill

Jonathan Lebensold

Collaborateur·rice alumni - McGill

Collaborateur·rice alumni - McGill

jeremy.lucas@mila.quebec

Ray Luo

Doctorat - McGill

Superviseur⋅e principal⋅e :

Xujie Si

luo.ziyan@mila.quebec

Sicheng Lyu

Maîtrise recherche - McGill

sicheng.lyu@mila.quebec

G McCracken

Doctorat - McGill

mccrackg@mila.quebec

shahrad.mohammadzadeh@mila.quebec

Shahrad Mohammadzadeh

Maîtrise recherche - McGill

Superviseur⋅e principal⋅e :

Reihaneh Rabbany

padideh.nouri@mila.quebec

Gabriela Moisescu-Pareja

Collaborateur·rice de recherche - McGill

Co-superviseur⋅e :

Irina Rish

moisescg@mila.quebec

Padideh Nouri

Doctorat - UdeM

Doctorat - McGill

Co-superviseur⋅e :

Nate Rahn

Doctorat - McGill

Superviseur⋅e principal⋅e :

Marc Gendron-Bellemare

nathan.rahn@mila.quebec

Mandana Samiei

Doctorat - McGill

Co-superviseur⋅e :

Blake Richards

samiemandana@gmail.com

Doctorat - McGill

Co-superviseur⋅e :

Paul Masset

manoosh.samiei@mila.quebec

Doctorat - McGill

Nishanth Anand Vemgal

Doctorat - McGill

Doctorat - McGill

Co-superviseur⋅e :

Samira Ebrahimi Kahou

Stagiaire de recherche - McGill

guangyuan.wang@mila.quebec

Zihan Wang

Doctorat - McGill

zihan.wang@mila.quebec

william.potter@mila.quebec

Steve Wen

Maîtrise recherche - McGill

Co-superviseur⋅e :

Gregory Dudek

steve.wen@mila.quebec

Will William

Stagiaire de recherche - McGill

shuyuan.zhang@mila.quebec

Zijing Wu

Doctorat - McGill

Superviseur⋅e principal⋅e :

Paul Masset

zijing.wu@mila.quebec

Doctorat - McGill

Harry Zhao

Collaborateur·rice alumni - McGill

Co-superviseur⋅e :

Billets de blogue

Generic thumbnail for Mila Blog articles.

22 février 2024

Skipper : combiner l’abstraction spatiale et temporelle afin d’améliorer la généralisation

par

Mingde Harry Zhao

Safa Alver

Harm van Seijen

Romain Laroche

Doina Precup

Yoshua Bengio

Lire l'article

Publications

Rotation-Preserving Supervised Fine-Tuning

Mohammad Hamdaqa

Supervised fine-tuning (SFT) improves in-domain performance but can degrade out-of-domain (OOD) generalization. Prior work suggests that thi… (voir plus)s degradation is related to changes in dominant singular subspaces of pretrained weight matrices. However, directly identifying loss-sensitive directions with Hessian or Fisher information is computationally expensive at LLM scale. In this work, we propose preserving projected rotations in pretrained singular subspaces as an efficient proxy for Fisher-sensitive directions, which we call Rotation-Preserving Supervised Fine-Tuning (RPSFT). RPSFT penalizes changes in the projected top-

2026-12-30

OpenReview (publié)

Learning Where to Constrain: Meta-Learned Trust Allocation for Mathematical Reasoning

Guangyuan Wang

Padideh Nouri

Long mathematical solutions expose a weakness in uniform policy-update constraints. Early token changes affect a long suffix, while late cha… (voir plus)nges affect fewer future states. Prefix divergence-based methods like CPPO encode this asymmetry with a fixed position schedule and prefix budget. We introduce Meta-TA, a bilevel method for learning both choices from post-update performance on disjoint prompts. A direct one-step design fails when behavior and current policies match because the constraint starts inactive. We prove the resulting hypergradient equals zero. A detached warm-up creates controlled policy drift. A soft-gated step then supplies the meta-gradient. On Qwen3-8B-Base, five frozen-retraining seeds raise mean AIME24/25/26 Avg@16 from 29.01 for CPPO to 31.95 for Meta-TA with closely matched realized KL. At a 16k response cap, Meta-TA lowers normalized reasoning effective horizon from 0.475 to 0.430 and importance-ratio variance from 0.0534 to 0.0475. These results link adaptive trust allocation to higher accuracy and more localized updates in long-horizon mathematical reasoning.

2026-10-03

MATH-AI @ Neural Information Processing Systems (poster)

Exploration-Preserving Policy Optimization

Hangzhan Jin

Mohammad Hamdaqa

Reinforcement learning with verifiable rewards improves reasoning, while the allocation of learning signal shapes which solutions remain acc… (voir plus)essible under repeated sampling. Group-relative objectives assign equal advantages to equally rewarded responses, making aggregate credit proportional to sampled mode frequency. We introduce Exploration-Preserving Policy Optimization (ExPPO), a lightweight advantage-shaping rule that redistributes credit using prompt-relative, length-normalized response surprisal and prompt pass rate. ExPPO combines bounded shaping with shared normalization to preserve verifier polarity and approximately maintain each prompt group's total absolute sequence-advantage mass. Our analysis characterizes response-level credit allocation alongside sampled mode updates, deriving local conditions for gains in entropy and correct-mode discovery. Experiments show improved in-domain and out-of-domain reasoning coverage, higher aggregate response accuracy, and strong coverage at large sampling budgets. A controlled multi-answer evaluation further demonstrates increased correct-mode yield and gains in diversity among verified-correct responses. Code is available at https://github.com/jinhangzhan/ExPPO

2026-10-01

arXiv (prépublication)

Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning

Élodie Coté-Gauthier

Isabeau Prémont-Schwarz

A reinforcement-learning agent moving between tasks in a fixed environment faces a mismatch: the reward changes, but the geometry does not. … (voir plus)Methods that learn a representation and a value function jointly relearn both at every task switch, which is slow and prone to forgetting. We take the identification of grid cells with eigenvectors of the successor representation as a design principle, and build the feature basis from the low-frequency eigenvectors of the symmetrised random-walk graph Laplacian, defined over state-action pairs rather than states so that the features encode what an action does. Because this basis diagonalises the transition operator, the successor-feature transform is exactly diagonal with closed-form entries fixed by the Laplacian eigenvalues and the discount. No temporal-difference learning is needed for it, and neither a successor-feature network nor a replay buffer is required: the only learned quantity is a linear reward map, re-estimated per task. On two goal locations in a two-room maze sharing an identical transition structure, the resulting agent holds near-ceiling return across every task switch, where a successor-feature baseline that learns its features collapses at each change. The cost is a slightly lower ceiling imposed by the linear parameterisation. The basis here is computed from the true transition graph, so this arm is an oracle upper bound rather than a method competing on equal information; learning the basis online is in progress.

2026-09-29

WiML @ Neural Information Processing Systems (publié)

From Connectivity to Rewards: Dense Reward Learning with Directed State Graphs

Shuyuan Zhang

Zihan Wang

Xiao-Wen Chang

The integration of graphs with Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL) has received increasing attention, as graphs nat… (voir plus)urally encode task hierarchies for effective subgoal sampling. However, existing methods often overlook intrinsic connectivity information, failing to fully leverage the underlying topology for efficient learning. Most graph-based GCHRL methods use the graph as a stochastic sampling tool rather than as an environmental model that encodes connectivity and state-accessibility information. This limitation is particularly acute in quasimetric environments, where the inherent asymmetry of state transitions poses a fundamental challenge to stable policy learning and robust path planning. In this paper, we address these problems by introducing a state connectivity model designed to predict pairwise state connectivity strength in asymmetric environments. We transform these connectivity strengths into scalar auxiliary dense rewards, providing continuous guidance across multiple hierarchical levels. We demonstrate that our proposed framework, Graph-Guided Quasimetric Dense Reward (G2QDR), can theoretically be integrated into any existing GCHRL architecture, and the state connectivity model is efficiently implemented via a neural network trained on a directed state graph generated during exploration. Empirical results across a wide range of sparse reward environments indicate that, in general, G2QDR can enhance the performance of baseline GCHRL approaches with acceptable computational overhead.

2026-09-07

Transactions on Machine Learning Research (accepté)

SIC-Agents: Benchmarking and Building an Adaptive Simulator for Pediatric Serious Illness Communication Training

Zihan Wang

Anita Marie Slominska

Rennie Bimman

Elizabeth Di Flumeri

Amanda Mayappo-Neeposh

Conall Francoeur

Tamara Ellen Carver

Xiao-Wen Chang

Esin Darici Haritaoglu

Ismail Haritaoglu

Akshatha Arodi

Naomi Goloff

Pediatric serious illness communication (SIC) is critically important, yet scalable communication training for clinicians remains limited. C… (voir plus)ompared with other dialogue simulation settings, pediatric SIC poses additional challenges, including multi-party interactions, response to parental distress and strong dependence on feedback dynamics. Existing LLM-based simulators optimize generic dialogue quality rather than curriculum-contingent behavior required for effective SIC training. In collaboration with educators and pediatric clinicians, we introduce the first benchmark suite and simulation framework tailored to pediatric SIC training. Our benchmarks, PitfallBench and DialogueBench, evaluate simulators both at the turn-level and across full dialogues. We further propose SIC-Agents, a self-improving framework that generates a clinician-editable skill document to guide simulator behavior. Our experiments show that SIC-Agents outperforms static expert prompting. To support future research, we release our benchmarks for parent simulation in pediatric SIC at https://github.com/Beikewzh/sic-benchmarks

2026-08-29

arXiv (prépublication)

Analytic Planning under Uncertainty with Moment Closure

Shishir Sharma

Effective model-based reinforcement learning in stochastic environments requires planning that accounts for predictive uncertainty. Propagat… (voir plus)ing full state distributions analytically offers a principled way to do this, but has traditionally required restrictive policy or reward structures to remain tractable. Consequently, modern deep reinforcement learning has largely retreated to either stochastic sampling, which introduces significant target variance, or deterministic point estimates that ignore predictive covariance entirely. We investigate whether distribution-aware planning is possible without these constraints. Using a quadratic action-value parameterization, we first reduce the Bellman backup to an expectation over the state-value function alone; the key idea is then a compatibility principle between the predictive transition distribution and the value function class, under which this expectation is analytic in the distribution's moments. We instantiate this principle with a Gaussian transition model paired with a radial-basis value function, yielding a closed-form backup that propagates both predictive mean and covariance. Empirically, our approach reduces target variance and yields well-calibrated predictive uncertainty under stochastic observations in continuous control, providing a principled framework for planning with learned distribution models.

2026-08-02

arXiv (prépublication)

Addressing Sparse-Rewards in RL with Scalable Hierarchical Novel Eigen Options

Isabeau Prémont-Schwarz

Temporally extended exploration via graph Laplacian-based options is a promising approach to sparse-reward reinforcement learning (RL), but … (voir plus)existing methods either do not explicitly target novelty or fail to scale to pixel-based domains under function approximation. Novel Exploration via Orthogonality (NEO) addresses the first issue by constructing options that navigate from highly visited regions toward less visited ones, yet prior results were limited to settings where exact eigenvectors can be computed. We present a scalable extension of NEO to pixel-based domains, built on three contributions. First, we use a novelty-weighted continuous Laplacian graph-drawing objective, which enables RL with continuous observations. Second, we embed the resulting eigen-potential options within a hierarchical reinforcement learning framework, enabling coherent temporally extended behavior. Third, we observe that learned eigen-potential rewards are directional but locally unreliable under online approximation; we therefore augment each option reward with a novelty bonus, a novel design idea that proves essential for stabilizing option learning while preserving novelty-directed exploration. Together, these contributions yield stronger and more persistent exploration, enabling longer option rollouts and better access to hard-to-reach novel states. Empirically, our method significantly outperforms both the prior scalable Laplacian-option baseline and a direct extension of NEO on sparse-reward benchmarks under a fixed budget. On Montezuma's Revenge, our best variant achieves approximately 1.8x higher return than both baselines. On Venture, both baselines yield returns near zero, whereas our method achieves a return of 1135. Across seven hard ProcGen games, our method achieves approximately 3.5x and 5.6x higher aggregate normalized return than the two baselines, respectively.

2026-07-20

RLBRew @ Reinforcement Learning Conference (publié)

Toward a general understanding of neural representations learned by deep neural networks on group multiplications

Arthur Ayestas Hilgert

Sihui Wei

Gabriela Moisescu-Pareja

Gavin McCracken

We study the neural representations learned by deep neural networks trained on alternating group multiplication, discovering that they are S… (voir plus)chreier coset graphs, a generalization of Cayley graphs. Previous works inspected neural representations learned from cyclic group multiplication and identified them as Cayley graphs. Since cyclic group multiplication is Abelian, all subgroups are normal, and Schreier coset graphs reduce to Cayley graphs in this case. This contribution makes a step toward finding a general theory of what neural representations are in networks learning group multiplications.

2026-07-15

Topology, Algebra, and Geometry in Data Science (présentation orale)

To Retain or to Adapt? Generalizing Continual Learning

Giulia Lanzillotta

Mandana Samiei

Razvan Pascanu

Claire Vernade

The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting. This objective rests on a per… (voir plus)vasive, often unstated assumption: that a lifelong learner should approximate the Joint-Task Learning (JTL) solution and retain all previously acquired knowledge. We challenge this retention-centered premise, arguing that in non-stationary environments prioritizing retention can impede real-time adaptation. Shifting the focus to the Average Lifelong Error (ALE), we formalize CL as an online optimization problem governed by the interaction between environmental and learning dynamics. We introduce Transfer Efficiency as a quantitative measure of the tension between Instability, the bias inherited from conflicting past experience, and Transient Error, the optimization cost of learning new tasks from scratch. Under mild convergence conditions, holding across linear and neural network models, this decomposition yields a Critical Task Duration: a closed-form threshold beyond which historical knowledge transitions from a warm-start advantage to an optimization liability whenever retention induces a positive stationary bias. We validate these theoretical predictions on continual image classification and reinforcement learning benchmarks. Finally, by connecting continual learning to the online learning framework of predictable sequences, we show that JTL is only one instance of a broader family of objectives, and we propose a new general class of continual learning algorithms, which we call Predictive Continual Learning. Predictive CL algorithms optimize expected future performance under an explicit, dynamically updated model of future tasks. As a proof of concept, we analyze a Window algorithm that interpolates between JTL and Independent-Task Learning (ITL), outperforming both under controlled distributional drift.

2026-07-05

arXiv (prépublication)

Adaptive Multi-Horizon Reinforcement Learning

Manoosh Samiei

Paul Masset

Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences. In reinforcement le… (voir plus)arning (RL), this trade-off is typically controlled through a fixed discount factor, which imposes a single exponentially discounted temporal horizon. However, biological agents exhibit flexible and adaptive temporal discounting, suggesting that effective planning requires multiple timescales. Here, we propose a multi-horizon approach that adaptively selects and combines temporal horizons, enabling robust adaptation to changes in reward structure without manual discount-factor tuning. This flexibility makes the method particularly suitable for continual learning scenarios involving task switches and varying environmental configurations. Empirically, we demonstrate that our approach identifies effective discount factors across a range of MiniGrid environments, including continual settings composed of three sequentially changing tasks. These results suggest that adaptive temporal discounting can improve parameter efficiency and enhance adaptability in both artificial and biologically inspired learning systems.

2026-06-15

ContinualRL @ Reinforcement Learning Conference (publié)

A Bellman Optimality Equation for Plasticity

Jeremy Lucas

In continual reinforcement learning, carefully managing the stability-plasticity tradeoff remains a core challenge. Recent work by Abel et a… (voir plus)l. (2025) formalized this dilemma by defining plasticity as the generalized directed information from an agent’s observations to its actions, and empowerment as the generalized directed information from its actions to its observations. This formulation successfully reframes the traditional stability-plasticity tradeoff as an empowerment-plasticity tradeoff. However, while extensive literature exists on optimizing for empowerment, there is currently no research addressing the optimization of plasticity under this new definition. This paper presents preliminary work toward optimizing plasticity within Markov decision processes. We show that there exists a Bellman optimality equation for optimizing plasticity similar to previous work for empowerment.

2026-06-15

ContinualRL @ Reinforcement Learning Conference (publié)