Glen Berseth

alexandre.brown@mila.quebec

Biography

Glen Berseth is an assistant professor in the Department of Computer Science and Operations Research (DIRO) at Université de Montréal and a core academic member of Mila – Quebec Artificial Intelligence Institute.

He is a Canada CIFAR AI Chair and co-directs the Robotics and Embodied AI Lab (REAL). He was formerly a postdoctoral researcher at Berkeley Artificial Intelligence Research (BAIR), working with Sergey Levine.

Berseth’s previous and current research has focused on solving sequential decision-making problems (planning) for real-world autonomous learning systems (robots). More specifically, his research has focused on human-robot collaboration, reinforcement, and continual-, meta-, multi-agent and hierarchical learning.

He has published in the top venues in robotics, machine learning and computer animation. He teaches a course on robot learning at Université de Montréal and at Mila, in which he covers the most recent research on machine learning techniques for creating generalist robots.

Current Students

Özgür Aslan

PhD - Université de Montréal

ozgur.aslan@mila.quebec

PhD - Université de Montréal

Master's Research - Université de Montréal

charlotte.cloutier@mila.quebec

Roger Creus-Castanyer

PhD - Université de Montréal

Co-supervisor :

Pablo Samuel Castro

roger.creus-castanyer@mila.quebec

PhD - McGill University

Principal supervisor :

elham.daneshmand@mila.quebec

Léa Demeule

PhD - Université de Montréal

Co-supervisor :

Chris Pal

lea.demeule@mila.quebec

homayoun.honari@mila.quebec

Homayoun Honari

Collaborating researcher - Université de Montréal

Naser Kazemi

Collaborating researcher - EPFL

naser.kazemi@mila.quebec

Adriana Knatchbull-Hugessen

PhD - Université de Montréal

adriana.knatchbull-hugessen@mila.quebec

artur.kuramshin@mila.quebec

Artur Kuramshin

Master's Research - Université de Montréal

daniel.lawson@mila.quebec

Daniel Lawson

PhD - Université de Montréal

Co-supervisor :

Khimya Khetarpal

Postdoctorate - Université de Montréal

tabitha-edith.lee@mila.quebec

PhD - Université de Montréal

faisal.mohamed@mila.quebec

Research Intern - Université de Montréal

josuesmjr.mongan@gmail.com

michael.przystupa@mila.quebec

Michael Przystupa

Postdoctorate - Université de Montréal

Esra'a Saleh

PhD - Université de Montréal

Co-supervisor :

Aaron Courville

esraa.saleh@mila.quebec

jesse.silverberg@mila.quebec

Jesse Silverberg

PhD - Université de Montréal

Co-supervisor :

Marc Gendron-Bellemare

PhD - Université de Montréal

Co-supervisor :

Yoshua Bengio

siddarth.venkatraman@mila.quebec

Real-time Reinforcement Learning

Albert Zhan

PhD - Université de Montréal

albert.zhan@mila.quebec

Blog Posts

Deux robots dans une cuisine, en train de préparer le dîner. L'un coupe les légumes et l'autre fait une omelette.

June 20, 2025

Ivan Anokhin

Matthew Riemer

Rishav Rishav

Gopeshh Subbaraj

Glen Berseth

Read the article

February 15, 2022

Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation Primary tabs View Edit(active tab) Delete Revisions

Jędrzej Orbik

Charles Sun

Coline Devin

Glen Berseth

Read the article

Publications

SafeExplorer: An Unbiased Mixed-Policy Gradient for Reinforcement Learning with Recovery Interventions

Elham Daneshmand

Majid Khadiv

Training agents directly on the task they will be deployed on often leads to the best policy performance, but training on physical robots ca… (see more)n cause damage through falls, so minimizing training falls is critical. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region. The resulting mixed-policy rollouts bias the on-policy update, and importance sampling cannot remove this bias when the recovery policy is deterministic. We propose SafeExplorer, a drop-in modification of proximal policy optimization (PPO) built on an unbiased policy-gradient estimator of the mixed-policy return. The estimator uses the score function only inside the safe region and never evaluates the recovery policy's density, so deterministic recovery policies pose no problem. On final reward, the estimator matches or outperforms truncated importance sampling, which is itself biased for that same gradient. SafeExplorer also speeds up learning near the safe-region boundary with a closed-form critic target at recovery-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only from successful recoveries. On three simulated locomotion environments with ten seeds each, and given an already available recovery controller, SafeExplorer incurs

2026-10-17

Transactions on Machine Learning Research (accepted)

Imagining Goals with Geometric Horizon Models for Policy Generalization

Adriana Hugessen

Training from offline data has allowed for substantial progress in domains such as robotics, leading to general-purpose policies that can be… (see more) easily applied zero-shot or efficiently finetuned for downstream tasks. However, these policies can still have poor generalization, both due to the choice of modeling objective and from learning from a static dataset. In this work, we focus on the challenging task of zero-shot goal generalization, where a policy is evaluated on unseen tasks that require combining its existing knowledge (compositional generalization). An avenue for improving a policy's generalization is by generating new experience with world models; however, such generation has proven difficult for longer horizons. Thus, to alleviate this issue, we propose TD-Aug, which samples from a geometric horizon model and allows for directly imagining novel outcomes that can be achieved by composing existing knowledge. We demonstrate that training on these future outcomes as goals for goal-conditioned BC policies significantly improves generalization in stitching-based OGBench tasks.

2026-10-12

arXiv (preprint)

Gated Transformer World Models for Model-Based Reinforcement Learning

Model-based reinforcement learning has emerged as a promising approach for training sample-efficient policies by learning compact latent dyn… (see more)amics models that simulate, plan, and act through imagination. Recent work has shown that transformer-based world models outperform recurrent architectures due to their ability to capture long-range temporal dependencies. However, standard self-attention mechanisms apply the attention score uniformly across all dimensions of the token's representation, lacking an explicit mechanism to regulate information flow across dimensions. We introduce a gating mechanism for transformer-based world models that implements a feature-wise modulation mechanism within the attention computation. The gating module adaptively scales dimensions in a data-dependent way, enabling the model to selectively control contextual information. Our gating mechanism achieves greater sample-efficient policy learning through normalized mean performance improvements for the Atari 100k benchmark when applied to the STORM and TWISTER transformer world models.

2026-10-02

WM_PAI @ Advances in Neural Information Processing Systems (poster)

GlyphBench: A Playground for Language-Model Reinforcement Learning

Roger Creus Castanyer

Marc-Alexandre Côté

Matthew James Sargent

Augustine N. Mavor-Parker

Pablo Samuel Castro

We introduce GlyphBench, an environment suite for reinforcement learning (RL) post-training of language-model agents, with over 360 tasks sp… (see more)anning diverse games. GlyphBench renders spatial observations as two-dimensional Unicode grids and connects training, evaluation, and trajectory replay through a unified interface designed to support efficient and reproducible research. We use GlyphBench to study how observation interfaces, reasoning effort, and agent harnesses affect performance, and how RL configurations shape learning dynamics. Our results show that glyph observations outperform native text and pixels in our Craftax experiments, with further gains on several BALROG environments. RL on 100 GlyphBench tasks improves Qwen3.5-4B on held-out Reasoning Gym problems, reaching 63.48% accuracy and outperforming the base model, a math-trained baseline, and a code-trained baseline. These experiments provide empirical evidence that reasoning gains from gameplay can yield stronger transfer than math or code. Together, these results highlight GlyphBench's value as a testbed for systematic research on how language-model agents learn, interact, and generalize.

2026-09-27

arXiv (preprint)

KV-streams for Efficient Compaction in Agentic Reinforcement Learning

Emiliano Penaloza

Dane Malenfant

Dheeraj Vattikonda

Roger Creus Castanyer

Siddarth Venkatraman

Abhay Puri

Jonathan Light

Matthew J. Sargent

Augustine N. Mavor-Parker

M. Caccia

Lucas Caccia

Esmeralda S. Whitammer

Minseon Kim

Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been… (see more) the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.

2026-09-27

arXiv (preprint)

KV-streams for Efficient Compaction in Agentic Reinforcement Learning

Emiliano Penaloza

Dane Malenfant

Dheeraj Vattikonda

Roger Creus Castanyer

Siddarth Venkatraman

Abhay Puri

Jonathan Light

Matthew J. Sargent

Augustine N. Mavor-Parker

M. Caccia

Lucas Caccia

Esmeralda S. Whitammer

Minseon Kim

2026-09-27

arXiv (preprint)

Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotion

Chenhao Li

Marco Hutter

World models promise a paradigm shift in robotics, where an agent learns the physics of its environment once and then acquires behaviors eff… (see more)iciently. Yet the learned dynamics models at their core are typically morphology locked. In legged locomotion, a dynamics model trained on an ANYmal-D quadruped fails on a Unitree Go1 because it overfits to one robot's embodiment rather than capturing the locomotion dynamics shared across robots, so even a small change in actuator dynamics or limb length forces retraining from scratch. However, if we formalize a robot's unique physical traits into a morphology specification, a controller for a family of robots can utilize this blueprint in two ways. It can feed the specification to a model-free policy, or it can feed the specification to a learned dynamics model and extract the policy in imagination. We argue for the second route and introduce the Quadrupedal World Model (QWM), which conditions a single generative dynamics model on scale-invariant physical features and trains policies entirely inside it, through a physical morphology encoder, an adaptive reward normalizer, and morphology conditioning in the latent dynamics. Holding the morphology information identical, a model-free policy matches QWM on the training cohort but degrades on unseen morphologies, while QWM transfers zero-shot with no fine-tuning, adaptation, or warm-up in such cases. To our knowledge, this is the first world model to demonstrate zero-shot cross-embodiment transfer within the quadrupedal family.

2026-09-03

Conference on Robot Learning (poster)

CURATE: Automatic Curriculum Learning for Reinforcement Learning Agents through Competence-Based Curriculum Policy Search in Structured Task Spaces

Tabitha Edith Lee

Nan Rosemary Ke

Sarvesh Patil

Annya Dahmani

Eunice Yiu

Esra'a Saleh

Alison Gopnik

Oliver Kroemer

Due to fundamental exploration challenges without informed priors or specialized algorithms, agents may be unable to consistently receive in… (see more)formative rewards, leading to inefficient or intractable learning. To address these challenges, we introduce CURATE, an automatic curriculum learning algorithm for reinforcement learning agents in structured task spaces of monotonic difficulty. Through "exploration by exploitation," CURATE dynamically scales the task difficulty to match the agent's current competence. By exploiting its current capabilities that were learned in easier tasks, the agent improves its exploration in more difficult tasks. Our key insight is that the learning improvement in tasks that are close to those used for training is inversely proportional to their difficulty, and an agent that chooses a nearby distribution of the easiest unsolved tasks at any given time can automatically induce an easiest-to-hardest curriculum in these task spaces. To achieve this, CURATE conducts policy search in the task space to learn the best task distribution for training. As the agent's mastery grows, the learned curriculum adapts in an approximately easiest-to-hardest and task-directed fashion, efficiently culminating in a performant agent. Our experiments across three diverse domains (MiniGrid, Procgen, BipedalWalker) demonstrate that CURATE learns effective curricula for sample efficiency and proficiency with the potential for yielding broadly capable agents, matching or exceeding prior curriculum methods that do not require informed initialization or predefined schedules.

2026-08-30

Transactions on Machine Learning Research (accepted)

Imagining Goals for Policy Generalization with Geometric Horizon Models

Adriana Hugessen

Training from offline data has allowed for substantial progress in domains such as robotics, leading to general-purpose policies that can be… (see more) easily applied zero-shot or efficiently finetuned for downstream tasks. However, training policies on offline data can lead to poor generalization, both due to the choice of modeling objective and from learning from a static dataset. In this work, we focus on the challenging task of zero-shot goal generalization, where a policy is evaluated on unseen tasks that require reusing its existing knowledge (compositional generalization). An avenue for improving a policy's generalization is through generating new experience through world models; however, such generation has proven difficult for longer horizons. Thus, to alleviate this issue, we propose TD-Aug, sampling from a geometric horizon model, which allows for directly imagining novel outcomes that can be achieved through composing existing knowledge. We demonstrate that training on these future outcomes as goals for goal-conditioned BC and offline RL policies improves generalization in stitching-based OGBench tasks.

2026-07-20

RLBRew @ Reinforcement Learning Conference (published)

SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions

Elham Daneshmand

Majid Khadiv

Training reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot … (see more)be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrained Markov decision process (MDP) formulations do. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region (a subset of state space it should stay within), but the resulting mixed-policy rollouts silently bias every on-policy update, and the importance-sampling correction that would remove this bias is ill-defined whenever the recovery policy is deterministic. We address this bias with a drop-in modification of proximal policy optimization (PPO). Its core is an unbiased policy-gradient estimator that uses the score function only at safe timesteps and never evaluates the recovery policy's density, so it stays valid even when the recovery policy is deterministic, exactly where importance sampling breaks, and it empirically dominates importance sampling even when the recovery policy is stochastic. Because the recovery policy still makes credit assignment slow near the safe-region boundary, two further components accelerate learning: a closed-form value for recovery-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only when recovery succeeds. On a three-environment, five-seed benchmark, the resulting algorithm reduces training-time falls by factors of 233x, 48x, and 26x on HalfCheetah, Ant, and Unitree Go1 over standard PPO, while matching or exceeding PPO's final reward, and on Ant, where the recovery policy is unreliable, it is the only method that reaches 80% of the best final reward.

2026-07-08

arXiv (preprint)

SLowRL: Safe Low-Rank Adaptation for Bridging the Sim-to-Real Gap in Legged Locomotion

Elham Daneshmand

Shafeef Omar

Majid Khadiv

A simulator is, at best, a coarse low-fidelity model of the real world the agent eventually has to act in. Closing this residual gap on hard… (see more)ware is a canonical instance of operating in a big world: the real environment exposes contact dynamics, latencies, and disturbances that the agent was never given the capacity (parameters or data) to model during pretraining. Naive on-hardware fine-tuning is risky --- the policy can damage the robot before it improves --- and full-parameter updates require prohibitive interaction time. We propose SLowRL, a continual fine-tuning framework that confronts this big-world adaptation problem with two complementary forms of capacity limitation: (i) a rank-1 LoRA adapter applied per layer to both actor and critic, restricting each layer's update to a single direction in its image space (

2026-06-09

rl-conference.cc/RLC/2026/Workshop/RL_in_Big_Worlds (poster)

AI Agent Safety is a Reinforcement Learning Problem

Reginald McLean

Tabitha Edith Lee

Montaser Mohammedalamen

Kevin Roice

Patrick M. Pilarski

Marlos C. Machado

Alyssa Lefaivre Škopac

Benjamin Rosman

With the rapid advancement and deployment of Agentic AI, our scientific understanding of capabilities and limitations has not kept pace, lea… (see more)ding to cases where AI agents cause harm. We argue that many of these safety limitations are not novel problems. Instead, the safety challenges currently facing AI agents can be seen as instances of problems the reinforcement learning (RL) community has studied rigorously for decades. The core of this argument concerns the problem formulation of AI agents. AI agents are designed to solve sequential decision-making problems: problems with long-term objectives in which actions have delayed consequences. To model these types of problem, the problem is set up the problem such that the agent receives observations, feedback on its progress, and then takes actions. This is precisely the formulation of the RL problem. In this paper, we formalize the problem equivalence, which we then leverage to argue that \textbf{AI Agent safety is a reinforcement learning problem: the failure modes currently observed in deployed AI agents are structural instances of problems RL has formalized for decades, and the RL safety literature provides principled tools to diagnose and address them.}. We conclude with a call for deliberate collaboration between the RL and AI agent research communities: AI agent researchers gain access to principled frameworks, while RL researchers gain a class of real-world problems that could expose fundamental gaps in current RL benchmarks and theory.

2026-05-22

AIWILD @ International Conference on Machine Learning (published)