Doina Precup

Samin Yeasar Arnob

PhD - McGill University

arnobsam@mila.quebec

sumana.basu@mail.mcgill.ca

Sumana Basu

Collaborating Alumni - McGill University

Co-supervisor :

Adriana Romero Soriano

Master's Research - McGill University

gunbir-singh.baveja@mila.quebec

Collaborating Alumni - McGill University

cheluver@mila.quebec

Raymond Chua

PhD - McGill University

Co-supervisor :

PhD - McGill University

Principal supervisor :

David Meger

chungwes@mila.quebec

Jonathan Colaço Carr

Master's Research - McGill University

Principal supervisor :

Prakash Panangaden

jonathan.colaco-carr@mila.quebec

Élodie Coté-Gauthier

Master's Research - McGill University

Co-supervisor :

Isabeau Prémont-Schwarz

elodie.cote-gauthier@mila.quebec

Franco Del Balso

Master's Research - McGill University

franco.delbalso@mila.quebec

busra-tugce.gurbuz@mila.quebec

Jesse Farebrother

PhD - McGill University

Principal supervisor :

Marc Gendron-Bellemare

PhD - McGill University

Principal supervisor :

Eilif B. Muller

Mohammad Sami Nur Islam Islam

Jonathan Hu

Master's Research - McGill University

jonathan.hu@mila.quebec

Howard Huang

PhD - McGill University

huanghow@mila.quebec

Master's Research - McGill University

mohammad-sami-nur.islam@mila.quebec

Hangzhan Jin

PhD - Polytechnique Montréal

jinhangzhan@gmail.com

flemming.kondrup@mila.quebec

Flemming Kondrup

Postdoctorate - McGill University

Jonathan Lebensold

Collaborating Alumni - McGill University

Collaborating Alumni - McGill University

jeremy.lucas@mila.quebec

Ray Luo

PhD - McGill University

Principal supervisor :

Xujie Si

luo.ziyan@mila.quebec

Sicheng Lyu

Master's Research - McGill University

sicheng.lyu@mila.quebec

G McCracken

PhD - McGill University

mccrackg@mila.quebec

shahrad.mohammadzadeh@mila.quebec

Shahrad Mohammadzadeh

Master's Research - McGill University

Principal supervisor :

Reihaneh Rabbany

padideh.nouri@mila.quebec

Gabriela Moisescu-Pareja

Collaborating researcher - McGill University

Co-supervisor :

Irina Rish

moisescg@mila.quebec

Padideh Nouri

PhD - Université de Montréal

PhD - McGill University

Co-supervisor :

Research Intern - McGill University

william.potter@mila.quebec

manoosh.samiei@mila.quebec

Nate Rahn

PhD - McGill University

Principal supervisor :

Marc Gendron-Bellemare

nathan.rahn@mila.quebec

Mandana Samiei

PhD - McGill University

Co-supervisor :

Blake Richards

samiemandana@gmail.com

PhD - McGill University

Co-supervisor :

Paul Masset

PhD - McGill University

Nishanth Anand Vemgal

PhD - McGill University

PhD - McGill University

Co-supervisor :

Samira Ebrahimi Kahou

Zihan Wang

PhD - McGill University

zihan.wang@mila.quebec

guangyuan.wang@mila.quebec

Guangyuan Wang

Research Intern - McGill University

Steve Wen

Master's Research - McGill University

Co-supervisor :

Gregory Dudek

steve.wen@mila.quebec

Zijing Wu

PhD - McGill University

Principal supervisor :

Paul Masset

zijing.wu@mila.quebec

PhD - McGill University

shuyuan.zhang@mila.quebec

Harry Zhao

Collaborating Alumni - McGill University

Co-supervisor :

Blog Posts

Generic thumbnail for Mila Blog articles.

February 22, 2024

Skipper: Combining Spatial and Temporal Abstraction for Better Generalization

Mingde Harry Zhao

Safa Alver

Harm van Seijen

Romain Laroche

Doina Precup

Yoshua Bengio

Read the article

Publications

Attention Option-Critic

Raviteja Chunduru

2020-07-12

ICML.cc/2020/Workshop/LifelongML (accepted)

openreview.net

A Brief Look at Generalization in Visual Meta-Reinforcement Learning

Safa Alver

Due to the realization that deep reinforcement learning algorithms trained on high-dimensional tasks can strongly overfit to their training … (see more)environments, there have been several studies that investigated the generalization performance of these algorithms. However, there has been no similar study that evaluated the generalization performance of algorithms that were specifically designed for generalization, i.e. meta-reinforcement learning algorithms. In this paper, we assess the generalization performance of these algorithms by leveraging high-dimensional, procedurally generated environments. We find that these algorithms can display strong overfitting when they are evaluated on challenging tasks. We also observe that scalability to high-dimensional tasks with sparse rewards remains a significant problem among many of the current meta-reinforcement learning algorithms. With these results, we highlight the need for developing meta-reinforcement learning algorithms that can both generalize and scale.

2020-07-12

ICML.cc/2020/Workshop/LifelongML (accepted)

openreview.net

SVRG for Policy Evaluation with Fewer Gradient Evaluations

Stochastic variance-reduced gradient (SVRG) is an optimization method originally designed for tackling machine learning problems with a fini… (see more)te sum structure. SVRG was later shown to work for policy evaluation, a problem in reinforcement learning in which one aims to estimate the value function of a given policy. SVRG makes use of gradient estimates at two scales. At the slower scale, SVRG computes a full gradient over the whole dataset, which could lead to prohibitive computation costs. In this work, we show that two variants of SVRG for policy evaluation could significantly diminish the number of gradient calculations while preserving a linear convergence speed. More importantly, our theoretical result implies that one does not need to use the entire dataset in every epoch of SVRG when it is applied to policy evaluation with linear function approximation. Our experiments demonstrate large computational savings provided by the proposed methods.

2020-07-10

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (published)

Learning to Prove from Synthetic Theorems

Eser Aygün

Vlad Firoiu

Laurent Orseau

Shibl Mourad

A major challenge in applying machine learning to automated theorem proving is the scarcity of training data, which is a key ingredient in t… (see more)raining successful deep learning models. To tackle this problem, we propose an approach that relies on training with synthetic theorems, generated from a set of axioms. We show that such theorems can be used to train an automated prover and that the learned prover transfers successfully to human-generated theorems. We demonstrate that a prover trained exclusively on synthetic theorems can solve a substantial fraction of problems in TPTP, a benchmark dataset that is used to compare state-of-the-art heuristic provers. Our approach outperforms a model trained on human-generated problems in most axiom sets, thereby showing the promise of using synthetic data for this task.

2020-06-18

ArXiv (preprint)

A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms

Philip Amortila

Prakash Panangaden

Bellemare Marc-Emmanuel

We present a distributional approach to theoretical analyses of reinforcement learning algorithms for constant step-sizes. We demonstrate it… (see more)s effectiveness by presenting simple and unified proofs of convergence for a variety of commonly-used methods. We show that value-based methods such as TD(

2020-06-02

Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (published)

Efficient Planning under Partial Observability with Unnormalized Q Functions and Spectral Learning

2020-06-02

Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (published)

Value Preserving State-Action Abstractions

David Abel

Nathan Umbanhowar

Khimya Khetarpal

Dilip Arumugam

Michael L. Littman

2020-06-02

Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (published)

Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments

Martin Weiss

Simon Chamorro

Roger Girgis

Margaux Luck

Samira E. Kahou

Joseph P. Cohen

Millions of blind and visually-impaired (BVI) people navigate urban environments every day, using smartphones for high-level path-planning a… (see more)nd white canes or guide dogs for local information. However, many BVI people still struggle to travel to new places. In our endeavor to create a navigation assistant for the BVI, we found that existing Reinforcement Learning (RL) environments were unsuitable for the task. This work introduces SEVN, a sidewalk simulation environment and a neural network-based approach to creating a navigation agent. SEVN contains panoramic images with labels for house numbers, doors, and street name signs, and formulations for several navigation tasks. We study the performance of an RL algorithm (PPO) in this setting. Our policy model fuses multi-modal observations in the form of variable resolution images, visible text, and simulated GPS data to navigate to a goal door. We hope that this dataset, simulator, and experimental results will provide a foundation for further research into the creation of agents that can assist members of the BVI community with outdoor navigation.

2020-05-11

Proceedings of the Conference on Robot Learning (published)

Option-Critic in Cooperative Multi-Agent Systems

Jhelum Chakravorty

Nadeem Ward

Julien Roy

Maxime Chevalier-Boisvert

Sumana Basu

Andrei Lupu

In this paper, we investigate learning temporal abstractions in cooperative multi-agent systems, using the options framework (Sutton et al, … (see more)1999). First, we address the planning problem for the decentralized POMDP represented by the multi-agent system, by introducing a \emph{common information approach}. We use the notion of \emph{common beliefs} and broadcasting to solve an equivalent centralized POMDP problem. Then, we propose the Distributed Option Critic (DOC) algorithm, which uses centralized option evaluation and decentralized intra-option improvement. We theoretically analyze the asymptotic convergence of DOC and build a new multi-agent environment to demonstrate its validity. Our experiments empirically show that DOC performs competitively against baselines and scales with the number of agents.

2020-05-04

International Joint Conference on Autonomous Agents and Multiagent Systems (published)

Algorithmic Improvements for Deep Reinforcement Learning applied to Interactive Fiction

Vishal Jain

William Fedus

Hugo Larochelle

Bellemare Marc-Emmanuel

Text-based games are a natural challenge domain for deep reinforcement learning algorithms. Their state and action spaces are combinatoriall… (see more)y large, their reward function is sparse, and they are partially observable: the agent is informed of the consequences of its actions through textual feedback. In this paper we emphasize this latter point and consider the design of a deep reinforcement learning agent that can play from feedback alone. Our design recognizes and takes advantage of the structural characteristics of text-based games. We first propose a contextualisation mechanism, based on accumulated reward, which simplifies the learning problem and mitigates partial observability. We then study different methods that rely on the notion that most actions are ineffectual in any given situation, following Zahavy et al.'s idea of an admissible action. We evaluate these techniques in a series of text-based games of increasing difficulty based on the TextWorld framework, as well as the iconic game Zork. Empirically, we find that these techniques improve the performance of a baseline deep reinforcement learning agent applied to text-based games.

2020-04-02

Proceedings of the AAAI Conference on Artificial Intelligence (published)

Gifting in Multi-Agent Reinforcement Learning (Student Abstract)

Andrei Lupu

This work performs a first study on multi-agent reinforcement learning with deliberate reward passing between agents. We empirically demonst… (see more)rate that such mechanics can greatly improve the learning progression in a resource appropriation setting and provide a preliminary discussion of the complex effects of gifting on the learning dynamics.

2020-04-02

AAAI Conference on Artificial Intelligence (published)

Options of Interest: Temporal Abstraction with Interest Functions

Khimya Khetarpal

Martin Klissarov

Maxime Chevalier-Boisvert

Pierre-Luc Bacon

Temporal abstraction refers to the ability of an agent to use behaviours of controllers which act for a limited, variable amount of time. Th… (see more)e options framework describes such behaviours as consisting of a subset of states in which they can initiate, an internal policy and a stochastic termination condition. However, much of the subsequent work on option discovery has ignored the initiation set, because of difficulty in learning it from data. We provide a generalization of initiation sets suitable for general function approximation, by defining an interest function associated with an option. We derive a gradient-based learning algorithm for interest functions, leading to a new interest-option-critic architecture. We investigate how interest functions can be leveraged to learn interpretable and reusable temporal abstractions. We demonstrate the efficacy of the proposed approach through quantitative and qualitative results, in both discrete and continuous environments.

2020-04-02

Proceedings of the AAAI Conference on Artificial Intelligence (published)