Publications

Exploration-Driven Representation Learning in Reinforcement Learning

Akram Erraqabi

Harry Zhao

Mingde Zhao

Marlos C. Machado

Yoshua Bengio

Sainbayar Sukhbaatar

Ludovic Denoyer

Alessandro Lazaric

Learning reward-agnostic representations is an emerging paradigm in reinforcement learning. These representations can be leveraged for sever… (see more)al purposes ranging from reward shaping to skill discovery. Nevertheless, in order to learn such representations, existing methods often rely on assuming uniform access to the state space. With such a privilege, the agent’s coverage of the environment can be limited which hurts the quality of the learned representations. In this work, we introduce a method that explicitly couples representation learning with exploration when the agent is not provided with a uniform prior over the state space. Our method learns representations that constantly drive exploration while the data generated by the agent’s exploratory behavior drives the learning of better representations. We empirically validate our approach in goal-achieving tasks, demonstrating that the learned representation captures the dynamics of the environment, leads to more accurate value estimation, and to faster credit assignment, both when used for control and for reward shaping. Finally, the exploratory policy that emerges from our approach proves to be successful at continuous navigation tasks with sparse rewards.

2021-07-21

ICML.cc/2021/Workshop/URL (poster)

openreview.net

Diffusion magnetic resonance imaging reveals tract‐specific microstructural correlates of electrophysiological impairments in non‐myelopathic and myelopathic spinal cord compression

Jan Valosek

René Labounek

Tomáš Horák

Magda Horáková

Petr Bednařík

Miloš Keřkovský

Jan Kočica

Tomáš Rohan

Christophe Lenglet

Julien Cohen‐Adad

Petr Hluštı́k

Eva Vlčková

Zdeněk Kadaňka

Josef Bednařík

Alena Svátková

Non‐myelopathic degenerative cervical spinal cord compression (NMDC) frequently occurs throughout aging and may progress to potentially ir… (see more)reversible degenerative cervical myelopathy (DCM). Whereas standard clinical magnetic resonance imaging (MRI) and electrophysiological measures assess compression severity and neurological dysfunction, respectively, underlying microstructural deficits still have to be established in NMDC and DCM patients. The study aims to establish tract‐specific diffusion MRI markers of electrophysiological deficits to predict the progression of asymptomatic NMDC to symptomatic DCM.

2021-07-20

European Journal of Neurology (published)

doi.org

Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?

Dinghuai Zhang

Kartik Ahuja

Yilun Xu

Yisen Wang

Aaron Courville

Can models with particular structure avoid being biased towards spurious correlation in out-of-distribution (OOD) generalization? Peters et … (see more)al. (2016) provides a positive answer for linear cases. In this paper, we use a functional modular probing method to analyze deep model structures under OOD setting. We demonstrate that even in biased models (which focus on spurious correlation) there still exist unbiased functional subnetworks. Furthermore, we articulate and demonstrate the functional lottery ticket hypothesis: full network contains a subnetwork that can achieve better OOD performance. We then propose Modular Risk Minimization to solve the subnetwork selection problem. Our algorithm learns the subnetwork structure from a given dataset, and can be combined with any other OOD regularization methods. Experiments on various OOD generalization tasks corroborate the effectiveness of our method.

2021-07-19

International Conference on Machine Learning (Oral)

doi.org

proceedings.mlr.press

Continuous Coordination As a Realistic Scenario for Lifelong Learning

Hadi Nekoei

Akilesh Badrinaaraayanan

Aaron Courville

Sarath Chandar

Current deep reinforcement learning (RL) algorithms are still highly task-specific and lack the ability to generalize to new environments. L… (see more)ifelong learning (LLL), however, aims at solving multiple tasks sequentially by efficiently transferring and using knowledge between tasks. Despite a surge of interest in lifelong RL in recent years, the lack of a realistic testbed makes robust evaluation of LLL algorithms difficult. Multi-agent RL (MARL), on the other hand, can be seen as a natural scenario for lifelong RL due to its inherent non-stationarity, since the agents' policies change over time. In this work, we introduce a multi-agent lifelong learning testbed that supports both zero-shot and few-shot settings. Our setup is based on Hanabi -- a partially-observable, fully cooperative multi-agent game that has been shown to be challenging for zero-shot coordination. Its large strategy space makes it a desirable environment for lifelong RL tasks. We evaluate several recent MARL methods, and benchmark state-of-the-art LLL algorithms in limited memory and computation regimes to shed light on their strengths and weaknesses. This continual learning paradigm also provides us with a pragmatic way of going beyond centralized training which is the most commonly used training protocol in MARL. We empirically show that the agents trained in our setup are able to coordinate well with unseen agents, without any additional assumptions made by previous works. The code and all pre-trained models are available at https://github.com/chandar-lab/Lifelong-Hanabi.

2021-07-19

International Conference on Machine Learning (Spotlight)

doi.org

proceedings.mlr.press

A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation

Scott Fujimoto

David Meger

Doina Precup

Marginalized importance sampling (MIS), which measures the density ratio between the state-action occupancy of a target policy and that of a… (see more) sampling distribution, is a promising approach for off-policy evaluation. However, current state-of-the-art MIS methods rely on complex optimization tricks and succeed mostly on simple toy problems. We bridge the gap between MIS and deep reinforcement learning by observing that the density ratio can be computed from the successor representation of the target policy. The successor representation can be trained through deep reinforcement learning methodology and decouples the reward optimization from the dynamics of the environment, making the resulting algorithm stable and applicable to high-dimensional domains. We evaluate the empirical performance of our approach on a variety of challenging Atari and MuJoCo environments.

2021-07-19

International Conference on Machine Learning (Spotlight)

doi.org

proceedings.mlr.press

Diffusion Earth Mover's Distance and Distribution Embeddings

Alexander Tong

Guillaume Huguet

Amine Natik

Kincaid MacDonald

Manik Kuchroo

Ronald R. Coifman

Guy Wolf

Smita Krishnaswamy

We propose a new fast method of measuring distances between large numbers of related high dimensional datasets called the Diffusion Earth Mo… (see more)ver's Distance (EMD). We model the datasets as distributions supported on common data graph that is derived from the affinity matrix computed on the combined data. In such cases where the graph is a discretization of an underlying Riemannian closed manifold, we prove that Diffusion EMD is topologically equivalent to the standard EMD with a geodesic ground distance. Diffusion EMD can be computed in

2021-07-19

International Conference on Machine Learning (Spotlight)

doi.org

proceedings.mlr.press

An End-to-End Framework for Molecular Conformation Generation via Bilevel Programming

Minkai Xu

Wujie Wang

Shitong Luo

Chence Shi

Yoshua Bengio

Rafael Gomez-Bombarelli

Jian Tang

Predicting molecular conformations (or 3D structures) from molecular graphs is a fundamental problem in many applications. Most existing app… (see more)roaches are usually divided into two steps by first predicting the distances between atoms and then generating a 3D structure through optimizing a distance geometry problem. However, the distances predicted with such two-stage approaches may not be able to consistently preserve the geometry of local atomic neighborhoods, making the generated structures unsatisfying. In this paper, we propose an end-to-end solution for molecular conformation prediction called ConfVAE based on the conditional variational autoencoder framework. Specifically, the molecular graph is first encoded in a latent space, and then the 3D structures are generated by solving a principled bilevel optimization program. Extensive experiments on several benchmark data sets prove the effectiveness of our proposed approach over existing state-of-the-art approaches. Code is available at https://github.com/MinkaiXu/ConfVAE-ICML21

2021-07-19

International Conference on Machine Learning (Spotlight)

doi.org

proceedings.mlr.press

Out-of-Distribution Generalization via Risk Extrapolation

David Krueger

Ethan Caballero

Joern-Henrik Jacobsen

Rémi Le Priol

Distributional shift is one of the major obstacles when transferring machine learning prediction systems from the lab to the real world. To … (see more)tackle this problem, we assume that variation across training domains is representative of the variation we might encounter at test time, but also that shifts at test time may be more extreme in magnitude. In particular, we show that reducing differences in risk across training domains can reduce a model's sensitivity to a wide range of extreme distributional shifts, including the challenging setting where the input contains both causal and anti-causal elements. We motivate this approach, Risk Extrapolation (REx), as a form of robust optimization over a perturbation set of extrapolated domains (MM-REx), and propose a penalty on the variance of training risks (V-REx) as a simpler variant. We prove that variants of REx can recover the causal mechanisms of the targets, while also providing some robustness to changes in the input distribution ("covariate shift"). By appropriately trading-off robustness to causally induced distributional shifts and covariate shift, REx is able to outperform alternative methods such as Invariant Risk Minimization in situations where these types of shift co-occur.

2021-07-19

International Conference on Machine Learning (Oral)

doi.org

proceedings.mlr.press

Preferential Temporal Difference Learning

Nishanth Anand

Doina Precup

Temporal-Difference (TD) learning is a general and very useful tool for estimating the value function of a given policy, which in turn is re… (see more)quired to find good policies. Generally speaking, TD learning updates states whenever they are visited. When the agent lands in a state, its value can be used to compute the TD-error, which is then propagated to other states. However, it may be interesting, when computing updates, to take into account other information than whether a state is visited or not. For example, some states might be more important than others (such as states which are frequently seen in a successful trajectory). Or, some states might have unreliable value estimates (for example, due to partial observability or lack of data), making their values less desirable as targets. We propose an approach to re-weighting states used in TD updates, both when they are the input and when they provide the target for the update. We prove that our approach converges with linear function approximation and illustrate its desirable empirical behaviour compared to other TD-style methods.

2021-07-19

International Conference on Machine Learning (Spotlight)

doi.org

proceedings.mlr.press

Randomized Exploration for Reinforcement Learning with General Value Function Approximation

Haque Ishfaq

Qiwen Cui

Viet Nguyen

Alex Ayoub

Zhuoran Yang

Zhaoran Wang

Doina Precup

Lin F. Yang

2021-07-19

International Conference on Machine Learning (Spotlight)

proceedings.mlr.press

Beyond Variance Reduction: Understanding the True Impact of Baselines on Policy Optimization

Wesley Chung

Valentin Thomas

Marlos C. Machado

Nicolas Le Roux

Bandit and reinforcement learning (RL) problems can often be framed as optimization problems where the goal is to maximize average performan… (see more)ce while having access only to stochastic estimates of the true gradient. Traditionally, stochastic optimization theory predicts that learning dynamics are governed by the curvature of the loss function and the noise of the gradient estimates. In this paper we demonstrate that this is not the case for bandit and RL problems. To allow our analysis to be interpreted in light of multi-step MDPs, we focus on techniques derived from stochastic optimization principles (e.g., natural policy gradient and EXP3) and we show that some standard assumptions from optimization theory are violated in these problems. We present theoretical results showing that, at least for bandit problems, curvature and noise are not sufficient to explain the learning dynamics and that seemingly innocuous choices like the baseline can determine whether an algorithm converges. These theoretical findings match our empirical evaluation, which we extend to multi-state MDPs.

2021-07-17

International Conference on Machine Learning (unknown)

doi.org

proceedings.mlr.press

Combating False Negatives in Adversarial Imitation Learning (Student Abstract)

Konrad Żołna

Chitwan Saharia

Léonard Boussioux

David Yu-tung Hui

Maxime Chevalier-Boisvert

Dzmitry Bahdanau

Yoshua Bengio