Doina Precup

busra-tugce.gurbuz@mila.quebec

Jesse Farebrother

Doctorat - McGill

Superviseur⋅e principal⋅e :

Marc Gendron-Bellemare

Doctorat - McGill

Superviseur⋅e principal⋅e :

Eilif B. Muller

Mohammad Sami Nur Islam Islam

Jonathan Hu

Maîtrise recherche - McGill

jonathan.hu@mila.quebec

Howard Huang

Doctorat - McGill

huanghow@mila.quebec

Maîtrise recherche - McGill

mohammad-sami-nur.islam@mila.quebec

Hangzhan Jin

Doctorat - Polytechnique

jinhangzhan@gmail.com

flemming.kondrup@mila.quebec

Flemming Kondrup

Postdoctorat - McGill

Jonathan Lebensold

Collaborateur·rice alumni - McGill

Collaborateur·rice alumni - McGill

jeremy.lucas@mila.quebec

Ray Luo

Doctorat - McGill

Superviseur⋅e principal⋅e :

Xujie Si

luo.ziyan@mila.quebec

Sicheng Lyu

Maîtrise recherche - McGill

sicheng.lyu@mila.quebec

G McCracken

Doctorat - McGill

mccrackg@mila.quebec

shahrad.mohammadzadeh@mila.quebec

Shahrad Mohammadzadeh

Maîtrise recherche - McGill

Superviseur⋅e principal⋅e :

Reihaneh Rabbany

padideh.nouri@mila.quebec

Gabriela Moisescu-Pareja

Collaborateur·rice de recherche - McGill

Co-superviseur⋅e :

Irina Rish

moisescg@mila.quebec

Padideh Nouri

Doctorat - UdeM

Doctorat - McGill

Co-superviseur⋅e :

Nate Rahn

Doctorat - McGill

Superviseur⋅e principal⋅e :

Marc Gendron-Bellemare

nathan.rahn@mila.quebec

Mandana Samiei

Doctorat - McGill

Co-superviseur⋅e :

Blake Richards

samiemandana@gmail.com

Doctorat - McGill

Co-superviseur⋅e :

Paul Masset

manoosh.samiei@mila.quebec

Doctorat - McGill

Nishanth Anand Vemgal

Doctorat - McGill

Doctorat - McGill

Co-superviseur⋅e :

Samira Ebrahimi Kahou

Stagiaire de recherche - McGill

guangyuan.wang@mila.quebec

Zihan Wang

Doctorat - McGill

zihan.wang@mila.quebec

william.potter@mila.quebec

Steve Wen

Maîtrise recherche - McGill

Co-superviseur⋅e :

Gregory Dudek

steve.wen@mila.quebec

Will William

Stagiaire de recherche - McGill

shuyuan.zhang@mila.quebec

Zijing Wu

Doctorat - McGill

Superviseur⋅e principal⋅e :

Paul Masset

zijing.wu@mila.quebec

Doctorat - McGill

Harry Zhao

Collaborateur·rice alumni - McGill

Co-superviseur⋅e :

Billets de blogue

Generic thumbnail for Mila Blog articles.

22 février 2024

Skipper : combiner l’abstraction spatiale et temporelle afin d’améliorer la généralisation

par

Mingde Harry Zhao

Safa Alver

Harm van Seijen

Romain Laroche

Doina Precup

Yoshua Bengio

Lire l'article

Publications

Perceived Regret: Evaluating Agents in Any World

Xiao-Wen Chang

Continual reinforcement learning considers an agent receiving and learning from a stream of experience. The agent only learns about its worl… (voir plus)d through these experiences and aims to maximize its accumulated reward. A fundamental problem is to evaluate this agent using only its stream of experience, without assuming a particular structure of the world. We outline a novel approach to this problem: Define an \textit{examiner} that observes the same stream of experience and, at every timestep, computes a \textit{perceived regret}, representing the agent's suboptimality gap in value from the examiner's perspective. We empirically demonstrate how perceived regret can be a useful performance measure, applicable to arbitrary streams of experience. Theoretically, we prove that there is no universal examiner, capable of accurately assessing all agents in any world but specific modeling biases can enable success in associated environments. Further experiments show the flexibility of this approach and explore its nuances.

2026-06-15

ContinualRL @ Reinforcement Learning Conference (publié)

QD-Learning for Continual Reinforcement Learning

Zijing Wu

Paul Masset

Nishanth Anand

In continual reinforcement learning (CRL), an agent continually adapts to an unbroken stream of experience by balancing two conflicting obje… (voir plus)ctives: stability and plasticity. To balance these two objectives, we introduce QD-learning, a new class of algorithms rooted in KL-regularized RL. QD-learning algorithms combine \textit{an adaptive default policy} with \textit{a soft-Q policy} to determine the agent's behaviour at each time step. Similar to habit learning in the brain, the adaptive default policy is updated to mimic the agent's overall behaviour, while the soft-Q policy is updated using TD-error along with a regularizing term based on the KL divergence between the default and the agent's current policies. Our paradigm, QD-learning, encompasses capacity-constrained RL and existing KL-regularized RL approaches as special cases while remaining general. We present three instantiations of QD-learning in dynamic programming and sample-based settings and analyze their theoretical properties via policy-improvement theorems. Additionally, we empirically evaluate our algorithms on both single-task and continual RL settings using a tabular chain MDP. The results show that QD-learning clearly outperforms baselines, particularly in environments with a large number of actions.

2026-06-15

ContinualRL @ Reinforcement Learning Conference (publié)

Hierarchical Integration of Predictive Representations of State from General Value Functions

Sonny Jones

Patrick M. Pilarski

Ashley N Dalrymple

In this work, we investigate how predictive representations of state in the form of continually learned General Value Functions (GVFs) inter… (voir plus)act with downstream policy networks. Intelligent agents deployed in real-world environments need to adapt to changing conditions in their environment. Adapting to one’s environment requires a model or representation of the environment on which to base decision-making. Models that take the form of predictions and GVFs have been shown to provide temporally abstracted predictive representations of state that can forecast useful elements of an agent's or environment's future behaviour. While GVFs have been concretely deployed in rehabilitation and robotic domains, existing approaches treat predictions as input features into model frameworks, without examining or comparing how best to integrate them into downstream learning processes. In this work, we compare multiple strategies for integrating observations and GVF predictions into another learning architecture: 1) actual observations solely in the input layer, 2) predictions solely in the input layer, 3) actual observations and predictions in the input layer, and 4) actual observations in the input layer and predictions in the later latent representations. We evaluate these strategies in a rehabilitation setting, using GVFs to learn predictive representations of kinetic and kinematic signals collected from wearable sensors on the lower limb during ambulation across varied terrains, and policy networks to classify walking terrain.

2026-06-09

rl-conference.cc/RLC/2026/Workshop/RL_in_Big_Worlds (poster)

Human Adults and LLMs as Scientists: Who Benefits from Active Exploration?

Mandana Samiei

Eunice Yiu

Anthony GX-Chen

Dongyan Lin

Jocelyn Shen

Blake A. Richards

Alison Gopnik

A long-standing finding in the causal learning literature is that adults struggle to identify conjunctive causal rules, where an effect requ… (voir plus)ires the simultaneous presence of multiple causes, while performing better in disjunctive settings. However, most demonstrations of this ``conjunctive handicap'' rely on passive observation paradigms with limited evidence, where learners have no control over evidence generation. This paper asks whether this bias persists when adults are granted agency through active exploration. Using a modified ``blicket detector'' task, adult participants freely intervened to identify causal objects under conjunctive or disjunctive rule structures. We show that active exploration substantially improves adults' conjunctive causal reasoning, although conjunctive rules still require more tests to infer than disjunctive rules. We further compare human performance to a range of large language models in the same setting. While some state-of-the-art models approach human-level performance on hypothesis inference accuracy, they often exhibit less efficient exploration strategies and similar conjunctive-disjunctive performance gaps.

2026-06-03

arXiv (prépublication)

Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning

Anthony GX-Chen

Ankit Anand

Gheorghe Comanici

Zaheer Abbas

Eser Aygün

David Smalling

Shibl Mourad

Andre Barreto

Mark Rowland

Classical reinforcement learning (RL) typically seeks a deterministic policy that maximizes the expected sum of a scalar reward. Yet, modern… (voir plus) applications such as language model fine-tuning or scientific discovery demand diversity. Existing remedies such as entropy regularization or diversity bonuses often require fragile trade-offs that sacrifice performance for stochasticity or rely on heuristic metrics that can misalign policy rankings. We argue that diversity is more naturally understood as the rational response to uncertainty in the reward. When the reward function is not perfectly known--as is the case with ambiguous preferences or imperfect reward models--committing to a single action can be sub-optimal. Building on this, we propose a fundamental reformulation of the RL objective by replacing the scalar reward with a distribution over reward functions, and applying a non-linear objective over sets of actions. The result is a framework in which calibrated behavioural diversity emerges naturally, remains controllable through the reward function distribution, and is obtained without sacrificing expected reward. Focusing on the contextual bandit setting, we derive a principled gradient estimator for this objective and prove that our formulation naturally generalizes both vanilla policy gradient and more recently developed action-set approaches. Our empirical results demonstrate that this framework offers a robust and theoretically grounded alternative for complex RL tasks where the traditional formulation of the problem fails to induce the desired breadth of agent behaviour.

2026-06-01

arXiv (prépublication)

The schema spectrum: Emergent structures and levels of abstraction in AI and the brain

Mandana Samiei

Blake A. Richards

2026-05-31

Neuron (publié)

A systematic review of human-LLM interactions in computational thinking empirical studies

Yimei Zhang

You Song

Reihaneh Rabbany

Maria Cutumisu

2026-05-11

Computer Science Education (publié)

Balancing Plasticity and Stability with Fast and Slow Successor Features

Raymond Chua

Blake Richards

A hallmark of intelligence is the ability to adapt in non-stationary environments, yet deep Reinforcement Learning (RL) agents often struggl… (voir plus)e in such settings. Prior studies introduce non-stationarity through abrupt shifts in features or dynamics, whereas real-world environments often evolve gradually through continual drift. This distinction has important implications for the ``stability-plasticity dilemma'' in RL, as abrupt task changes may demand more plasticity than naturalistic settings. To address this, we modify existing 3D Miniworld and MuJoCo environments to incorporate naturalistic, continual non-stationarity, and use them to examine how stability and adaptation affect performance under continuous environmental change. We find that methods favoring stability, such as synaptic consolidation, outperform approaches focused on plasticity, such as parameters resetting. Motivated by this result, and prior evidence that Successor Features (SFs) reduce interference, we investigate whether SFs are better consolidation targets than Q-values. Across both environments, applying neuro-inspired synaptic consolidation to SFs yields superior performance on continually changing settings. Moreover, consolidation is most effective when SFs are stabilized across multiple timescales, which capture complementary aspects of gradual environmental change. Together, these results suggest that stability is more critical in continual learning when changes are gradual, and that multi-timescale consolidation of predictive representations is an effective approach.

2026-04-29

International Conference on Machine Learning (accepté)

Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems

Jonathan Colaço Carr

Prakash Panangaden

Benjamin Van Roy

Reinforcement learning problems typically define the goal as maximizing the expected value of a scalar reward function. But, pairwise prefer… (voir plus)ences are often easier to specify than scalar rewards, and they express certain goals that scalar rewards cannot. Methods for reinforcement learning with pairwise preferences have thus received growing interest. Unfortunately, these methods are inefficient in problems with long time horizons, and they lack guarantees on the performance of Markov policies relative to history-dependent policies, which bridge the theory and practice of reinforcement learning. We therefore propose the \textit{Markov decision contest} as a new problem model for reinforcement learning with pairwise preferences. We prove that stationary Markov policies are optimal among all history-dependent policies, that solving a Markov decision contest exactly is in P, and that a simple iterative algorithm converges to an optimal policy at a sublinear rate. Lastly, in a set of high-dimensional decision problems with long time horizons, we show that our approximate algorithm is significantly more learning-efficient than prior work.

2026-04-29

International Conference on Machine Learning (accepté)

AIF-GEN: Open-Source Platform and Synthetic Dataset Suite for Reinforcement Learning on Large Language Models

Jacob Chmura

Shahrad Mohammadzadeh

Taz Scott-Talib

Nishanth Anand

2026-03-01

LLA @ International Conference on Learning Representations (poster)

Nazanin Mohammadi Sepahvand

Detoxifying LLMs via Representation Erasure-Based Preference Optimization

Eleni Triantafillou

Hugo Larochelle

Gintare Karolina Dziugaite

Daniel M. Roy

Large language models (LLMs) trained on webscale data can produce toxic outputs, raising concerns for safe deployment. Prior defenses, based… (voir plus) on applications of DPO, NPO, and similar algorithms, reduce the likelihood of harmful continuations, but not robustly so: they are vulnerable to adversarial prompting and easily undone by fine-tuning-based relearning attacks. Indeed, research has shown that these edits to the model are superficial: linear probing reveals that harmful "directions" remain present in representations. To address this, we propose Representation Erasure-based Preference Optimization (REPO), reformulating detoxification as a token-level preference problem. Using a novel objective with preference data, we force the representations of toxic continuations to converge toward their benign counterparts. Our mechanistic analysis reveals that this granular approach is critical: unlike baselines, REPO induces deep, localized edits to toxicity-encoding neurons while preserving general model utility. Exhaustive evaluations show that REPO achieves state-of-the-art robustness, stopping sophisticated threats-including relearning attacks and enhanced GCG jailbreaks-where existing representation- and output-based methods fail.

2026-02-23

arXiv (inconnu)

Fluid-Agent Reinforcement Learning

Shishir Sharma

Theodore J. Perkins

The primary focus of multi-agent reinforcement learning (MARL) has been to study interactions among a fixed number of agents embedded in an … (voir plus)environment. However, in the real world, the number of agents is neither fixed nor known a priori. Moreover, an agent can decide to create other agents (for example, a cell may divide, or a company may spin off a division). In this paper, we propose a framework that allows agents to create other agents; we call this a fluid-agent environment. We present game-theoretic solution concepts for fluid-agent games and empirically evaluate the performance of several MARL algorithms within this framework. Our experiments include fluid variants of established benchmarks such as Predator-Prey and Level-Based Foraging, where agents can dynamically spawn, as well as a new environment we introduce that highlights how fluidity can unlock novel solution strategies beyond those observed in fixed-population settings. We demonstrate that this framework yields agent teams that adjust their size dynamically to match environmental demands.

2026-02-15

ArXiv (prépublication)