Doina Precup

Sumana Basu

Collaborating Alumni - McGill University

Co-supervisor :

Adriana Romero Soriano

Collaborating Alumni - McGill University

Lynn Cherif

Collaborating Alumni - McGill University

Co-supervisor :

PhD - McGill University

Co-supervisor :

PhD - McGill University

Principal supervisor :

David Meger

Jonathan Colaço Carr

Master's Research - McGill University

Principal supervisor :

Prakash Panangaden

Élodie Coté-Gauthier

Collaborating researcher - McGill University

Co-supervisor :

Isabeau Prémont-Schwarz

Franco Del Balso

Collaborating researcher - Université de Montréal

Jesse Farebrother

PhD - McGill University

Principal supervisor :

Marc Gendron-Bellemare

PhD - McGill University

Principal supervisor :

Collaborating researcher - Birla Institute of Technology

Howard Huang

PhD - McGill University

Haque Ishfaq

Collaborating Alumni - McGill University

Mohammad Sami Nur Islam Islam

Master's Research - McGill University

Arushi Jain

Collaborating Alumni - McGill University

PhD - Polytechnique Montréal

Martin Klissarov

PhD - McGill University

Postdoctorate - McGill University

Jonathan Lebensold

Collaborating Alumni - McGill University

Collaborating Alumni - McGill University

Ray Luo

PhD - McGill University

Principal supervisor :

G McCracken

PhD - McGill University

Nazanin Mohammadi Sepahvand

Collaborating Alumni - McGill University

Shahrad Mohammadzadeh

Master's Research - McGill University

Principal supervisor :

Gabriela Moisescu-Pareja

Collaborating researcher - McGill University

Co-supervisor :

Irina Rish

Padideh Nouri

PhD - Université de Montréal

Co-supervisor :

PhD - McGill University

Co-supervisor :

Nate Rahn

PhD - McGill University

Principal supervisor :

Marc Gendron-Bellemare

Mandana Samiei

PhD - McGill University

Co-supervisor :

PhD - McGill University

Co-supervisor :

PhD - McGill University

Nishanth Anand Vemgal

PhD - McGill University

PhD - McGill University

Co-supervisor :

Samira Ebrahimi Kahou

Zihan Wang

PhD - McGill University

Skipper: Combining Spatial and Temporal Abstraction for Better Generalization

Guangyuan Wang

Research Intern - McGill University

Steve Wen

Master's Research - McGill University

Co-supervisor :

Gregory Dudek

Zijing Wu

PhD - McGill University

Principal supervisor :

PhD - McGill University

Harry Zhao

Collaborating Alumni - McGill University

Co-supervisor :

Blog Posts

Generic thumbnail for Mila Blog articles.

February 22, 2024

Mingde Harry Zhao

Safa Alver

Harm van Seijen

Romain Laroche

Doina Precup

Yoshua Bengio

Read the article

Publications

Policy Evaluation Networks

Jean Harb

Tom Schaul

Pierre-Luc Bacon

Many reinforcement learning algorithms use value functions to guide the search for better policies. These methods estimate the value of a si… (see more)ngle policy while generalizing across many states. The core idea of this paper is to flip this convention and estimate the value of many policies, for a single set of states. This approach opens up the possibility of performing direct gradient ascent in policy space without seeing any new data. The main challenge for this approach is finding a way to represent complex policies that facilitates learning and generalization. To address this problem, we introduce a scalable, differentiable fingerprinting mechanism that retains essential policy information in a concise embedding. Our empirical results demonstrate that combining these three elements (learned Policy Evaluation Network, policy fingerprints, gradient ascent) can produce policies that outperform those that generated the training data, in zero-shot manner.

2020-02-25

ArXiv (preprint)

oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions

David Venuto

Jhelum Chakravorty

Léonard Boussioux

Junhao Wang

Gavin McCracken

Explicit engineering of reward functions for given environments has been a major hindrance to reinforcement learning methods. While Inverse … (see more)Reinforcement Learning (IRL) is a solution to recover reward functions from demonstrations only, these learned rewards are generally heavily \textit{entangled} with the dynamics of the environment and therefore not portable or \emph{robust} to changing environments. Modern adversarial methods have yielded some success in reducing reward entanglement in the IRL setting. In this work, we leverage one such method, Adversarial Inverse Reinforcement Learning (AIRL), to propose an algorithm that learns hierarchical disentangled rewards with a policy over options. We show that this method has the ability to learn \emph{generalizable} policies and reward functions in complex transfer learning tasks, while yielding results in continuous control benchmarks that are comparable to those of the state-of-the-art methods.

2020-02-19

ArXiv (preprint)

Representation of Reinforcement Learning Policies in Reproducing Kernel Hilbert Spaces

Thang Doan

We propose a general framework for policy representation for reinforcement learning tasks. This framework involves finding a low-dimensional… (see more) embedding of the policy on a reproducing kernel Hilbert space (RKHS). The usage of RKHS based methods allows us to derive strong theoretical guarantees on the expected return of the reconstructed policy. Such guarantees are typically lacking in black-box models, but are very desirable in tasks requiring stability. We conduct several experiments on classic RL domains. The results confirm that the policies can be robustly embedded in a low-dimensional space while the embedded policy incurs almost no decrease in return.

2020-02-06

(published)

A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms

Philip Amortila

Prakash Panangaden

Bellemare Marc-Emmanuel

We present a distributional approach to theoretical analyses of reinforcement learning algorithms for constant step-sizes. We demonstrate it… (see more)s effectiveness by presenting simple and unified proofs of convergence for a variety of commonly-used methods. We show that value-based methods such as TD(

2019-12-31

AISTATS (published)

proceedings.mlr.press

On Efficiency in Hierarchical Reinforcement Learning

Zheng Wen

Morteza Ibrahimi

Andre Barreto

Benjamin Van Roy

Satinder Singh

Hierarchical Reinforcement Learning (HRL) approaches promise to provide more efﬁcient solutions to sequential decision making problems, bo… (see more)th in terms of statistical as well as computational efﬁciency. While this has been demonstrated empirically over time in a variety of tasks, theoretical results quantifying the ben-eﬁts of such methods are still few and far between. In this paper, we discuss the kind of structure in a Markov decision process which gives rise to efﬁcient HRL methods. Speciﬁcally, we formalize the intuition that HRL can exploit well repeating "subMDPs", with similar reward and transition structure. We show that, under reasonable assumptions, a model-based Thompson sampling-style HRL algorithm that exploits this structure is statistically efﬁcient, as established through a ﬁnite-time regret bound. We also establish conditions under which planning with structure-induced options is near-optimal and computationally efﬁcient.

2019-12-31

Advances in Neural Information Processing Systems 33 (NeurIPS 2020) (published)

dblp.uni-trier.de

Efficient Planning under Partial Observability with Unnormalized Q Functions and Spectral Learning

2019-12-31

International Conference on Artificial Intelligence and Statistics (published)

proceedings.mlr.press

An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay

Scott Fujimoto

David Meger

Prioritized Experience Replay (PER) is a deep reinforcement learning technique in which agents learn from transitions sampled with non-unifo… (see more)rm probability proportionate to their temporal-difference error. We show that any loss function evaluated with non-uniformly sampled data can be transformed into another uniformly sampled loss function with the same expected gradient. Surprisingly, we find in some environments PER can be replaced entirely by this new loss function without impact to empirical performance. Furthermore, this relationship suggests a new branch of improvements to PER by correcting its uniformly sampled loss function equivalent. We demonstrate the effectiveness of our proposed modifications to PER and the equivalent loss function in several MuJoCo and Atari environments.

2019-12-31

Advances in Neural Information Processing Systems 33 (NeurIPS 2020) (published)

Forethought and Hindsight in Credit Assignment

Veronica Chelu

Hado van Hasselt

We address the problem of credit assignment in reinforcement learning and explore fundamental questions regarding the way in which an agent … (see more)can best use additional computation to propagate new information, by planning with internal models of the world to improve its predictions. Particularly, we work to understand the gains and peculiarities of planning employed as forethought via forward models or as hindsight operating with backward models. We establish the relative merits, limitations and complementary properties of both planning mechanisms in carefully constructed scenarios. Further, we investigate the best use of models in planning, primarily focusing on the selection of states in which predictions should be (re)-evaluated. Lastly, we discuss the issue of model estimation and highlight a spectrum of methods that stretch from explicit environment-dynamics predictors to more abstract planner-aware models.

2019-12-31

Advances in Neural Information Processing Systems 33 (NeurIPS 2020) (published)

Reward Propagation Using Graph Convolutional Networks

Martin Klissarov

Potential-based reward shaping provides an approach for designing good reward functions, with the purpose of speeding up learning. However, … (see more)automatically finding potential functions for complex environments is a difficult problem (in fact, of the same difficulty as learning a value function from scratch). We propose a new framework for learning potential functions by leveraging ideas from graph representation learning. Our approach relies on Graph Convolutional Networks which we use as a key ingredient in combination with the probabilistic inference view of reinforcement learning. More precisely, we leverage Graph Convolutional Networks to perform message passing from rewarding states. The propagated messages can then be used as potential functions for reward shaping to accelerate learning. We verify empirically that our approach can achieve considerable improvements in both small and high-dimensional control problems.

2019-12-31

NeurIPS (published)

Reward Redistribution Mechanisms in Multi-agent Reinforcement Learning

Aly Ibrahim

Anirudha Jitani

Piracha

Daoud

In typical Multi-Agent Reinforcement Learning (MARL) settings, each agent acts to maximize its individual reward objective. However, for col… (see more)lective social welfare maximization, some agents may need to act non-selfishly. We propose a reward shaping mechanism using extrinsic motivation for achieving modularity and increased cooperation among agents in Sequential Social Dilemma (SSD) problems. Our mechanism, inspired by capitalism, provides extrinsic motivation to agents by redistributing a portion of collected re-wards based on each agent’s individual contribution towards team rewards. We demonstrate empirically that this mechanism leads to higher collective welfare relative to existing baselines. Furthermore, this reduces free rider issues and leads to more diverse policies. We evaluate our proposed mechanism for already specialised agents that are pre-trained for specific roles. We show that our mechanism, in the most challenging CleanUp environment, significantly out-performs two baselines (based roughly on socialism and anarchy) and accumulates 2-3 times higher rewards in an easier setting of the environment.

2019-12-31

(published)

www.semanticscholar.org

Value Preserving State-Action Abstractions

David Abel

Nathan Umbanhowar

Khimya Khetarpal

Dilip Arumugam

Michael L. Littman

2019-12-31

International Conference on Artificial Intelligence and Statistics (published)

proceedings.mlr.press

Value-driven Hindsight Modelling

Arthur Guez

Fabio Viola

Theophane Weber

Lars Buesing

Steven Kapturowski