Doina Precup

Mohammad Sami Nur Islam Islam

Jonathan Hu

Maîtrise recherche - McGill

jonathan.hu@mila.quebec

Howard Huang

Doctorat - McGill

huanghow@mila.quebec

Maîtrise recherche - McGill

mohammad-sami-nur.islam@mila.quebec

flemming.kondrup@mila.quebec

Hangzhan Jin

Doctorat - Polytechnique

jinhangzhan@gmail.com

Doctorat - McGill

Postdoctorat - McGill

Jonathan Lebensold

Collaborateur·rice alumni - McGill

Collaborateur·rice alumni - McGill

jeremy.lucas@mila.quebec

Ray Luo

Doctorat - McGill

luo.ziyan@mila.quebec

Site web

Sicheng Lyu

Maîtrise recherche - McGill

sicheng.lyu@mila.quebec

G McCracken

Doctorat - McGill

mccrackg@mila.quebec

shahrad.mohammadzadeh@mila.quebec

Shahrad Mohammadzadeh

Maîtrise recherche - McGill

Doctorat - UdeM

padideh.nouri@mila.quebec

Doctorat - McGill

Co-superviseur⋅e :

Stagiaire de recherche - McGill

william.potter@mila.quebec

Nate Rahn

Doctorat - McGill

nathan.rahn@mila.quebec

manoosh.samiei@mila.quebec

Doctorat - McGill

samiemandana@gmail.com

Doctorat - McGill

Doctorat - McGill

Nishanth Anand Vemgal

Doctorat - McGill

Doctorat - McGill

Co-superviseur⋅e :

Samira Ebrahimi Kahou

Zihan Wang

Doctorat - McGill

zihan.wang@mila.quebec

Site web

guangyuan.wang@mila.quebec

Guangyuan Wang

Stagiaire de recherche - McGill

Steve Wen

Maîtrise recherche - McGill

steve.wen@mila.quebec

Zijing Wu

Doctorat - McGill

zijing.wu@mila.quebec

Doctorat - McGill

shuyuan.zhang@mila.quebec

Billets de blogue

Generic thumbnail for Mila Blog articles.

22 février 2024

Skipper : combiner l’abstraction spatiale et temporelle afin d’améliorer la généralisation

par

Mingde Harry Zhao

Safa Alver

Harm van Seijen

Romain Laroche

Doina Precup

Yoshua Bengio

Lire l'article

Publications

Analytic Planning under Uncertainty with Moment Closure

Shishir Sharma

Effective model-based reinforcement learning in stochastic environments requires planning that accounts for predictive uncertainty. Propagat… (voir plus)ing full state distributions analytically offers a principled way to do this, but has traditionally required restrictive policy or reward structures to remain tractable. Consequently, modern deep reinforcement learning has largely retreated to either stochastic sampling, which introduces significant target variance, or deterministic point estimates that ignore predictive covariance entirely. We investigate whether distribution-aware planning is possible without these constraints. Using a quadratic action-value parameterization, we first reduce the Bellman backup to an expectation over the state-value function alone; the key idea is then a compatibility principle between the predictive transition distribution and the value function class, under which this expectation is analytic in the distribution's moments. We instantiate this principle with a Gaussian transition model paired with a radial-basis value function, yielding a closed-form backup that propagates both predictive mean and covariance. Empirically, our approach reduces target variance and yields well-calibrated predictive uncertainty under stochastic observations in continuous control, providing a principled framework for planning with learned distribution models.

2026-08-02

arXiv (prépublication)

Addressing Sparse-Rewards in RL with Scalable Hierarchical Novel Eigen Options

Isabeau Prémont-Schwarz

Temporally extended exploration via graph Laplacian-based options is a promising approach to sparse-reward reinforcement learning (RL), but … (voir plus)existing methods either do not explicitly target novelty or fail to scale to pixel-based domains under function approximation. Novel Exploration via Orthogonality (NEO) addresses the first issue by constructing options that navigate from highly visited regions toward less visited ones, yet prior results were limited to settings where exact eigenvectors can be computed. We present a scalable extension of NEO to pixel-based domains, built on three contributions. First, we use a novelty-weighted continuous Laplacian graph-drawing objective, which enables RL with continuous observations. Second, we embed the resulting eigen-potential options within a hierarchical reinforcement learning framework, enabling coherent temporally extended behavior. Third, we observe that learned eigen-potential rewards are directional but locally unreliable under online approximation; we therefore augment each option reward with a novelty bonus, a novel design idea that proves essential for stabilizing option learning while preserving novelty-directed exploration. Together, these contributions yield stronger and more persistent exploration, enabling longer option rollouts and better access to hard-to-reach novel states. Empirically, our method significantly outperforms both the prior scalable Laplacian-option baseline and a direct extension of NEO on sparse-reward benchmarks under a fixed budget. On Montezuma's Revenge, our best variant achieves approximately 1.8x higher return than both baselines. On Venture, both baselines yield returns near zero, whereas our method achieves a return of 1135. Across seven hard ProcGen games, our method achieves approximately 3.5x and 5.6x higher aggregate normalized return than the two baselines, respectively.

2026-07-20

RLBRew @ Reinforcement Learning Conference (publié)

Toward a general understanding of neural representations learned by deep neural networks on group multiplications

Arthur Ayestas Hilgert

Sihui Wei

Gabriela Moisescu-Pareja

Gavin McCracken

We study the neural representations learned by deep neural networks trained on alternating group multiplication, discovering that they are S… (voir plus)chreier coset graphs, a generalization of Cayley graphs. Previous works inspected neural representations learned from cyclic group multiplication and identified them as Cayley graphs. Since cyclic group multiplication is Abelian, all subgroups are normal, and Schreier coset graphs reduce to Cayley graphs in this case. This contribution makes a step toward finding a general theory of what neural representations are in networks learning group multiplications.

2026-07-15

Topology, Algebra, and Geometry in Data Science (présentation orale)

To Retain or to Adapt? Generalizing Continual Learning

Giulia Lanzillotta

Razvan Pascanu

Claire Vernade

The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting. This objective rests on a per… (voir plus)vasive, often unstated assumption: that a lifelong learner should approximate the Joint-Task Learning (JTL) solution and retain all previously acquired knowledge. We challenge this retention-centered premise, arguing that in non-stationary environments prioritizing retention can impede real-time adaptation. Shifting the focus to the Average Lifelong Error (ALE), we formalize CL as an online optimization problem governed by the interaction between environmental and learning dynamics. We introduce Transfer Efficiency as a quantitative measure of the tension between Instability, the bias inherited from conflicting past experience, and Transient Error, the optimization cost of learning new tasks from scratch. Under mild convergence conditions, holding across linear and neural network models, this decomposition yields a Critical Task Duration: a closed-form threshold beyond which historical knowledge transitions from a warm-start advantage to an optimization liability whenever retention induces a positive stationary bias. We validate these theoretical predictions on continual image classification and reinforcement learning benchmarks. Finally, by connecting continual learning to the online learning framework of predictable sequences, we show that JTL is only one instance of a broader family of objectives, and we propose a new general class of continual learning algorithms, which we call Predictive Continual Learning. Predictive CL algorithms optimize expected future performance under an explicit, dynamically updated model of future tasks. As a proof of concept, we analyze a Window algorithm that interpolates between JTL and Independent-Task Learning (ITL), outperforming both under controlled distributional drift.

2026-07-05

arXiv (prépublication)

Adaptive Multi-Horizon Reinforcement Learning

Manoosh Samiei

Paul Masset

Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences. In reinforcement le… (voir plus)arning (RL), this trade-off is typically controlled through a fixed discount factor, which imposes a single exponentially discounted temporal horizon. However, biological agents exhibit flexible and adaptive temporal discounting, suggesting that effective planning requires multiple timescales. Here, we propose a multi-horizon approach that adaptively selects and combines temporal horizons, enabling robust adaptation to changes in reward structure without manual discount-factor tuning. This flexibility makes the method particularly suitable for continual learning scenarios involving task switches and varying environmental configurations. Empirically, we demonstrate that our approach identifies effective discount factors across a range of MiniGrid environments, including continual settings composed of three sequentially changing tasks. These results suggest that adaptive temporal discounting can improve parameter efficiency and enhance adaptability in both artificial and biologically inspired learning systems.

2026-06-15

ContinualRL @ Reinforcement Learning Conference (publié)

A Bellman Optimality Equation for Plasticity

Jeremy Lucas

In continual reinforcement learning, carefully managing the stability-plasticity tradeoff remains a core challenge. Recent work by Abel et a… (voir plus)l. (2025) formalized this dilemma by defining plasticity as the generalized directed information from an agent’s observations to its actions, and empowerment as the generalized directed information from its actions to its observations. This formulation successfully reframes the traditional stability-plasticity tradeoff as an empowerment-plasticity tradeoff. However, while extensive literature exists on optimizing for empowerment, there is currently no research addressing the optimization of plasticity under this new definition. This paper presents preliminary work toward optimizing plasticity within Markov decision processes. We show that there exists a Bellman optimality equation for optimizing plasticity similar to previous work for empowerment.

2026-06-15

ContinualRL @ Reinforcement Learning Conference (publié)

Perceived Regret: Evaluating Agents in Any World

Xiao-Wen Chang

Continual reinforcement learning considers an agent receiving and learning from a stream of experience. The agent only learns about its worl… (voir plus)d through these experiences and aims to maximize its accumulated reward. A fundamental problem is to evaluate this agent using only its stream of experience, without assuming a particular structure of the world. We outline a novel approach to this problem: Define an \textit{examiner} that observes the same stream of experience and, at every timestep, computes a \textit{perceived regret}, representing the agent's suboptimality gap in value from the examiner's perspective. We empirically demonstrate how perceived regret can be a useful performance measure, applicable to arbitrary streams of experience. Theoretically, we prove that there is no universal examiner, capable of accurately assessing all agents in any world but specific modeling biases can enable success in associated environments. Further experiments show the flexibility of this approach and explore its nuances.

2026-06-15

ContinualRL @ Reinforcement Learning Conference (publié)

QD-Learning for Continual Reinforcement Learning

Zijing Wu

Paul Masset

Nishanth Anand

In continual reinforcement learning (CRL), an agent continually adapts to an unbroken stream of experience by balancing two conflicting obje… (voir plus)ctives: stability and plasticity. To balance these two objectives, we introduce QD-learning, a new class of algorithms rooted in KL-regularized RL. QD-learning algorithms combine \textit{an adaptive default policy} with \textit{a soft-Q policy} to determine the agent's behaviour at each time step. Similar to habit learning in the brain, the adaptive default policy is updated to mimic the agent's overall behaviour, while the soft-Q policy is updated using TD-error along with a regularizing term based on the KL divergence between the default and the agent's current policies. Our paradigm, QD-learning, encompasses capacity-constrained RL and existing KL-regularized RL approaches as special cases while remaining general. We present three instantiations of QD-learning in dynamic programming and sample-based settings and analyze their theoretical properties via policy-improvement theorems. Additionally, we empirically evaluate our algorithms on both single-task and continual RL settings using a tabular chain MDP. The results show that QD-learning clearly outperforms baselines, particularly in environments with a large number of actions.

2026-06-15

ContinualRL @ Reinforcement Learning Conference (publié)

Hierarchical Integration of Predictive Representations of State from General Value Functions

Sonny Jones

Patrick M. Pilarski

Ashley N Dalrymple

In this work, we investigate how predictive representations of state in the form of continually learned General Value Functions (GVFs) inter… (voir plus)act with downstream policy networks. Intelligent agents deployed in real-world environments need to adapt to changing conditions in their environment. Adapting to one’s environment requires a model or representation of the environment on which to base decision-making. Models that take the form of predictions and GVFs have been shown to provide temporally abstracted predictive representations of state that can forecast useful elements of an agent's or environment's future behaviour. While GVFs have been concretely deployed in rehabilitation and robotic domains, existing approaches treat predictions as input features into model frameworks, without examining or comparing how best to integrate them into downstream learning processes. In this work, we compare multiple strategies for integrating observations and GVF predictions into another learning architecture: 1) actual observations solely in the input layer, 2) predictions solely in the input layer, 3) actual observations and predictions in the input layer, and 4) actual observations in the input layer and predictions in the later latent representations. We evaluate these strategies in a rehabilitation setting, using GVFs to learn predictive representations of kinetic and kinematic signals collected from wearable sensors on the lower limb during ambulation across varied terrains, and policy networks to classify walking terrain.

2026-06-09

rl-conference.cc/RLC/2026/Workshop/RL_in_Big_Worlds (poster)

Human Adults and LLMs as Scientists: Who Benefits from Active Exploration?

Eunice Yiu

Anthony GX-Chen

Dongyan Lin

Jocelyn Shen

Blake A. Richards

Alison Gopnik

A long-standing finding in the causal learning literature is that adults struggle to identify conjunctive causal rules, where an effect requ… (voir plus)ires the simultaneous presence of multiple causes, while performing better in disjunctive settings. However, most demonstrations of this ``conjunctive handicap'' rely on passive observation paradigms with limited evidence, where learners have no control over evidence generation. This paper asks whether this bias persists when adults are granted agency through active exploration. Using a modified ``blicket detector'' task, adult participants freely intervened to identify causal objects under conjunctive or disjunctive rule structures. We show that active exploration substantially improves adults' conjunctive causal reasoning, although conjunctive rules still require more tests to infer than disjunctive rules. We further compare human performance to a range of large language models in the same setting. While some state-of-the-art models approach human-level performance on hypothesis inference accuracy, they often exhibit less efficient exploration strategies and similar conjunctive-disjunctive performance gaps.

2026-06-03

arXiv (prépublication)

Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning

Anthony GX-Chen

Ankit Anand

Gheorghe Comanici

Zaheer Abbas

Eser Aygün

David Smalling

Shibl Mourad

Andre Barreto

Mark Rowland

Classical reinforcement learning (RL) typically seeks a deterministic policy that maximizes the expected sum of a scalar reward. Yet, modern… (voir plus) applications such as language model fine-tuning or scientific discovery demand diversity. Existing remedies such as entropy regularization or diversity bonuses often require fragile trade-offs that sacrifice performance for stochasticity or rely on heuristic metrics that can misalign policy rankings. We argue that diversity is more naturally understood as the rational response to uncertainty in the reward. When the reward function is not perfectly known--as is the case with ambiguous preferences or imperfect reward models--committing to a single action can be sub-optimal. Building on this, we propose a fundamental reformulation of the RL objective by replacing the scalar reward with a distribution over reward functions, and applying a non-linear objective over sets of actions. The result is a framework in which calibrated behavioural diversity emerges naturally, remains controllable through the reward function distribution, and is obtained without sacrificing expected reward. Focusing on the contextual bandit setting, we derive a principled gradient estimator for this objective and prove that our formulation naturally generalizes both vanilla policy gradient and more recently developed action-set approaches. Our empirical results demonstrate that this framework offers a robust and theoretically grounded alternative for complex RL tasks where the traditional formulation of the problem fails to induce the desired breadth of agent behaviour.

2026-06-01

arXiv (prépublication)

The schema spectrum: Emergent structures and levels of abstraction in AI and the brain

Blake A. Richards

2026-05-31

Neuron (publié)