Glen Berseth

Biography

Glen Berseth is an assistant professor in the Department of Computer Science and Operations Research (DIRO) at Université de Montréal and a core academic member of Mila – Quebec Artificial Intelligence Institute.

He is a Canada CIFAR AI Chair and co-directs the Robotics and Embodied AI Lab (REAL). He was formerly a postdoctoral researcher at Berkeley Artificial Intelligence Research (BAIR), working with Sergey Levine.

Berseth’s previous and current research has focused on solving sequential decision-making problems (planning) for real-world autonomous learning systems (robots). More specifically, his research has focused on human-robot collaboration, reinforcement, and continual-, meta-, multi-agent and hierarchical learning.

He has published in the top venues in robotics, machine learning and computer animation. He teaches a course on robot learning at Université de Montréal and at Mila, in which he covers the most recent research on machine learning techniques for creating generalist robots.

Current Students

Özgür Aslan

PhD - Université de Montréal

Master's Research - Université de Montréal

Florence Cloutier

Professional Master's - Université de Montréal

Roger Creus-Castanyer

PhD - Université de Montréal

Co-supervisor :

PhD - McGill University

Principal supervisor :

Hsiu-Chin Lin

Léa Demeule

PhD - Université de Montréal

Co-supervisor :

Chris Pal

Charlie Gauthier

PhD - Université de Montréal

Principal supervisor :

Liam Paull

Victor Gilbert

Research Intern - Polytechnic

Adriana Knatchbull-Hugessen

Kumaraditya Gupta

Collaborating researcher

Principal supervisor :

PhD - Université de Montréal

Artur Kuramshin

Master's Research - Université de Montréal

Daniel Lawson

PhD - Université de Montréal

Co-supervisor :

Master's Research - Université de Montréal

Postdoctorate - Université de Montréal

Co-supervisor :

Professional Master's - Université de Montréal

Michael Przystupa

Research Intern - Université de Montréal

Esra'a Saleh

PhD - Université de Montréal

Co-supervisor :

Aaron Courville

Tang Tang

Postdoctorate - Université de Montréal

Siddarth Venkatraman

PhD - Université de Montréal

Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation Primary tabs View Edit(active tab) Delete Revisions

Blog Posts

February 15, 2022

Jędrzej Orbik

Charles Sun

Coline Devin

Glen Berseth

Read the article

Publications

Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching

Arnav Kumar Jain

Harley Wiltzer

Jesse Farebrother

Irina Rish

Sanjiban Choudhury

2024-11-11

ArXiv (preprint)

arxiv.org

Minimally Invasive Morphology Adaptation via Parameter Efficient Fine-Tuning

Michael Przystupa

Hongyao Tang

Mariano Phielipp

Santiago Miret

Martin Jägersand

Learning reinforcement learning policies to control individual robots is often computationally non-economical because minor variations in ro… (see more)bot morphology (e.g. dynamics or number of limbs) can negatively impact policy performance. This limitation has motivated morphology agnostic policy learning, in which a monolithic deep learning policy learns to generalize between robotic morphologies. Unfortunately, these policies still have sub-optimal zero-shot performance compared to end-to-end finetuning on target morphologies. This limitation has ramifications in practical robotic applications, as online finetuning large neural networks can require immense computation. In this work, we investigate \textit{parameter efficient finetuning} techniques to specialize morphology-agnostic policies to a target robot that minimizes the number of learnable parameters adapted during online learning. We compare direct finetuning, which update subsets of the base model parameters, and input-learnable approaches, which add additional parameters to manipulate inputs passed to the base model. Our analysis concludes that tuning relatively few parameters (0.01\% of the base model) can measurably improve policy performance over zero shot. These results serve a prescriptive purpose for future research for which scenarios certain PEFT approaches are best suited for adapting policy's to new robotic morphologies.

2024-10-23

corl.org/2024/Workshop/MAPoDeL (published)

Learning Robust Representations for Transfer in Reinforcement Learning

Faisal Mohamed

Roger Creus Castanyer

Hongyao Tang

Zahra Sheikhbahaee

Learning transferable representations for deep reinforcement learning (RL) is a challenging problem due to the inherent non-stationarity, di… (see more)stribution shift, and unstable training dynamics. To be useful, a transferable representation needs to be robust to such factors. In this work, we introduce a new architecture and training strategy for learning robust representations for transfer learning in RL. We propose leveraging multiple CNN encoders and training them not to specialize in areas of the state space but instead to match each other's representation. We find that learned representations transfer well across many Atari tasks, resulting in better transfer learning performance and data efficiency than training from scratch.

2024-10-10

NeurIPS.cc/2024/Workshop/FITML (poster)

Efficient Design-and-Control Automation with Reinforcement Learning and Adaptive Exploration

Jiajun Fan

Hongyao Tang

Michael Przystupa

Mariano Phielipp

Santiago Miret

Seeking good designs is a central goal of many important domains, such as robotics, integrated circuits (IC), medicine, and materials scienc… (see more)e. These design problems are expensive, time-consuming, and traditionally performed by human experts. Moreover, the barriers to domain knowledge make it challenging to propose a universal solution that generalizes to different design problems. In this paper, we propose a new method called Efficient Design and Stable Control (EDiSon) for automatic design and control in different design problems. The key ideas of our method are (1) interactive sequential modeling of the design and control process and (2) adaptive exploration and design replay. To decompose the difficulty of learning design and control as a whole, we leverage sequential modeling for both the design process and control process, with a design policy to generate step-by-step design proposals and a control policy to optimize the objective by operating the design. With deep reinforcement learning (RL), the policies learn to find good designs by maximizing a reward signal that evaluates the quality of designs. Furthermore, we propose an adaptive exploration and replay mechanism based on a design memory that maintains high-quality designs generated so far. By regulating between constructing a design from scratch or replaying a design from memory to refine it, EDiSon balances the trade-off between exploration and exploitation in the design space and stabilizes the learning of the control policy. In the experiments, we evaluate our method in robotic morphology design and Tetris-based design tasks. Our framework has the potential to significantly accelerate the discovery of optimized designs across diverse domains, including automated materials discovery, by improving the exploration in design space while ensuring efficiency.

2024-10-07

NeurIPS.cc/2024/Workshop/AI4Mat (published)

Amortizing intractable inference in diffusion models for vision, language, and control

Siddarth Venkatraman

Moksh J. Jain

Luca Scimeca

Minsu Kim

Marcin Sendera

Mohsin Hasan

Luke Rowe

Sarthak Mittal

Pablo Lemos

Emmanuel Bengio

Alexandre Adam

Jarrid Rector-Brooks

Yoshua Bengio

Nikolay Malkin

Diffusion models have emerged as effective distribution estimators in vision, language, and reinforcement learning, but their use as priors … (see more)in downstream tasks poses an intractable posterior inference problem. This paper studies amortized sampling of the posterior over data,

2024-09-25

NeurIPS.cc/2024/Conference (poster)

Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn

Hongyao Tang

Deep neural networks provide Reinforcement Learning (RL) powerful function approximators to address large-scale decision-making problems. Ho… (see more)wever, these approximators introduce challenges due to the non-stationary nature of RL training. One source of the challenges in RL is that output predictions can churn, leading to uncontrolled changes after each batch update for states not included in the batch. Although such a churn phenomenon exists in each step of network training, how churn occurs and impacts RL remains under-explored. In this work, we start by characterizing churn in a view of Generalized Policy Iteration with function approximation, and we discover a chain effect of churn that leads to a cycle where the churns in value estimation and policy improvement compound and bias the learning dynamics throughout the iteration. Further, we concretize the study and focus on the learning issues caused by the chain effect in different settings, including greedy action deviation in value-based methods, trust region violation in proximal policy optimization, and dual bias of policy value in actor-critic methods. We then propose a method to reduce the chain effect across different settings, called Churn Approximated ReductIoN (CHAIN), which can be easily plugged into most existing DRL algorithms. Our experiments demonstrate the effectiveness of our method in both reducing churn and improving learning performance across online and offline, value-based and policy-based RL settings, as well as a scaling setting.

2024-09-25

NeurIPS.cc/2024/Conference (poster)

Simplifying Constraint Inference with Inverse Reinforcement Learning

Adriana Hugessen

Harley Wiltzer

2024-09-25

NeurIPS.cc/2024/Conference (poster)

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

Alexander Khazatsky

Karl Pertsch

Suraj Nair

Ashwin Balakrishna

Sudeep Dasari

Siddharth Karamcheti

Soroush Nasiriany

Mohan Kumar Srirama

Lawrence Yunliang Chen

Kirsty Ellis

Peter David Fagan

Joey Hejna

Masha Itkina

Marion Lepert

Yecheng Jason Ma

Ye Ma

Patrick Tree Miller

Jimmy Wu

Suneel Belkhale

Shivin Dass … (see 80 more)

Huy Ha

Arhan Jain

Abraham Lee

Youngwoon Lee

Marius Memmel

Sungjae Park

Ilija Radosavovic

Kaiyuan Wang

Albert Zhan

Kevin Black

Cheng Chi

Kyle Beltran Hatch

Shan Lin

Jingpei Lu

Jean Mercat

Abdul Rehman

Pannag R Sanketi

Archit Sharma

Cody Simpson

Quan Vuong

Homer Rich Walke

Blake Wulfe

Ted Xiao

Jonathan Heewon Yang

Arefeh Yavary

Tony Z. Zhao

Christopher Agia

Rohan Baijal

Mateo Guaman Castro

Daphne Chen

Qiuyu Chen

Trinity Chung

Jaimyn Drake

Ethan Paul Foster

Jensen Gao

David Antonio Herrera

Minho Heo

Kyle Hsu

Jiaheng Hu

Donovon Jackson

Charlotte Le

Yunshuang Li

K. Lin

Roy Lin

Zehan Ma

Abhiram Maddukuri

Suvir Mirchandani

Daniel Morton

Tony Khuong Nguyen

Abigail O'Neill

Rosario Scalise

Derick Seale

Victor Son

Stephen Tian

Emi Tran

Andrew E. Wang

Yilin Wu

Annie Xie

Jingyun Yang

Patrick Yin

Yunchu Zhang

Osbert Bastani

Jeannette Bohg

Ken Goldberg

Abhinav Gupta

Abhishek Gupta

Dinesh Jayaraman

Joseph J Lim

Jitendra Malik

Roberto Martín-Martín

Subramanian Ramamoorthy

Dorsa Sadigh

Shuran Song

Jiajun Wu

Michael C. Yip

Yuke Zhu

Thomas Kollar

Sergey Levine

Chelsea Finn

The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and … (see more)robust robotic manipulation policies. However, creating such datasets is challenging: collecting robot manipulation data in diverse environments poses logistical and safety challenges and requires substantial investments in hardware and human labour. As a result, even the most general robot manipulation policies today are mostly trained on data collected in a small number of environments with limited scene and task diversity. In this work, we introduce DROID (Distributed Robot Interaction Dataset), a diverse robot manipulation dataset with 76k demonstration trajectories or 350 hours of interaction data, collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia, and Europe over the course of 12 months. We demonstrate that training with DROID leads to policies with higher performance and improved generalization ability. We open source the full dataset, policy learning code, and a detailed guide for reproducing our robot hardware setup.

2024-06-26

roboticsfoundation.org/RSS/2024/Workshop/DGR (poster)

Realtime Reinforcement Learning: Towards Rapid Asynchronous Deployment of Large Models

Matthew D Riemer

Gopeshh Subbaraj

Irina Rish

Realtime environments change even as agents perform action inference and learning, thus requiring high interaction frequencies to effectivel… (see more)y minimize long-term regret. However, recent advances in machine learning involve larger neural networks with longer inference times, raising questions about their applicability in realtime systems where reaction time is crucial. We present an analysis of lower bounds on regret in realtime environments to show that minimizing long-term regret is generally impossible within the typical sequential interaction and learning paradigm, but often becomes possible when sufficient asynchronous compute is available. We propose novel algorithms for staggering asynchronous inference processes to ensure that actions are taken at consistent time intervals, and demonstrate that use of models with high action inference times is only constrained by the environment's effective stochasticity over the inference horizon, and not by action frequency. Our analysis shows that the number of inference processes needed scales linearly with increasing inference times while enabling use of models that are multiple orders of magnitude larger than existing approaches when learning from a realtime simulation of Game Boy games such as Pokemon and Tetris.

2024-06-19

ICML.cc/2024/Workshop/ARLET (poster)

Revisiting Successor Features for Inverse Reinforcement Learning

Arnav Kumar Jain

Harley Wiltzer

Jesse Farebrother

Irina Rish

Sanjiban Choudhury

2024-06-17

ICML.cc/2024/Workshop/MFHAIA (poster)

RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning

Mingqi Yuan

Roger Creus Castanyer

Bo Li

Xin Jin

Wenjun Zeng

2024-05-29

ArXiv (preprint)

arxiv.org

Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learning

Adriana Hugessen

Roger Creus Castanyer

Faisal Mohamed

Both entropy-minimizing and entropy-maximizing (curiosity) objectives for unsupervised reinforcement learning (RL) have been shown to be eff… (see more)ective in different environments, depending on the environment's level of natural entropy. However, neither method alone results in an agent that will consistently learn intelligent behavior across environments. In an effort to find a single entropy-based method that will encourage emergent behaviors in any environment, we propose an agent that can adapt its objective online, depending on the entropy conditions by framing the choice as a multi-armed bandit problem. We devise a novel intrinsic feedback signal for the bandit, which captures the agent's ability to control the entropy in its environment. We demonstrate that such agents can learn to control entropy and exhibit emergent behaviors in both high- and low-entropy regimes and can learn skillful behaviors in benchmark tasks. Videos of the trained agents and summarized findings can be found on our project page https://sites.google.com/view/surprise-adaptive-agents

2024-05-14

rl-conference.cc/RLC/2024/Conference (published)