Publications

A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation
Scott Fujimoto
David Meger
Doina Precup
Marginalized importance sampling (MIS), which measures the density ratio between the state-action occupancy of a target policy and that of a… (voir plus) sampling distribution, is a promising approach for off-policy evaluation. However, current state-of-the-art MIS methods rely on complex optimization tricks and succeed mostly on simple toy problems. We bridge the gap between MIS and deep reinforcement learning by observing that the density ratio can be computed from the successor representation of the target policy. The successor representation can be trained through deep reinforcement learning methodology and decouples the reward optimization from the dynamics of the environment, making the resulting algorithm stable and applicable to high-dimensional domains. We evaluate the empirical performance of our approach on a variety of challenging Atari and MuJoCo environments.
2021-07-19
International Conference on Machine Learning (Spotlight)
doi.org
proceedings.mlr.press
Diffusion Earth Mover's Distance and Distribution Embeddings
Alexander Tong
Guillaume Huguet
Amine Natik
Kincaid MacDonald
Manik Kuchroo
Ronald R. Coifman
Guy Wolf
Smita Krishnaswamy
We propose a new fast method of measuring distances between large numbers of related high dimensional datasets called the Diffusion Earth Mo… (voir plus)ver's Distance (EMD). We model the datasets as distributions supported on common data graph that is derived from the affinity matrix computed on the combined data. In such cases where the graph is a discretization of an underlying Riemannian closed manifold, we prove that Diffusion EMD is topologically equivalent to the standard EMD with a geodesic ground distance. Diffusion EMD can be computed in
2021-07-19
International Conference on Machine Learning (Spotlight)
doi.org
proceedings.mlr.press
An End-to-End Framework for Molecular Conformation Generation via Bilevel Programming
Minkai Xu
Wujie Wang
Shitong Luo
Chence Shi
Yoshua Bengio
Rafael Gomez-Bombarelli
Jian Tang
Predicting molecular conformations (or 3D structures) from molecular graphs is a fundamental problem in many applications. Most existing app… (voir plus)roaches are usually divided into two steps by first predicting the distances between atoms and then generating a 3D structure through optimizing a distance geometry problem. However, the distances predicted with such two-stage approaches may not be able to consistently preserve the geometry of local atomic neighborhoods, making the generated structures unsatisfying. In this paper, we propose an end-to-end solution for molecular conformation prediction called ConfVAE based on the conditional variational autoencoder framework. Specifically, the molecular graph is first encoded in a latent space, and then the 3D structures are generated by solving a principled bilevel optimization program. Extensive experiments on several benchmark data sets prove the effectiveness of our proposed approach over existing state-of-the-art approaches. Code is available at https://github.com/MinkaiXu/ConfVAE-ICML21
2021-07-19
International Conference on Machine Learning (Spotlight)
doi.org
proceedings.mlr.press
Out-of-Distribution Generalization via Risk Extrapolation
David Krueger
Ethan Caballero
Joern-Henrik Jacobsen
Amy Zhang
Jonathan Binas
Dinghuai Zhang
Rémi Le Priol
Aaron Courville
Distributional shift is one of the major obstacles when transferring machine learning prediction systems from the lab to the real world. To … (voir plus)tackle this problem, we assume that variation across training domains is representative of the variation we might encounter at test time, but also that shifts at test time may be more extreme in magnitude. In particular, we show that reducing differences in risk across training domains can reduce a model's sensitivity to a wide range of extreme distributional shifts, including the challenging setting where the input contains both causal and anti-causal elements. We motivate this approach, Risk Extrapolation (REx), as a form of robust optimization over a perturbation set of extrapolated domains (MM-REx), and propose a penalty on the variance of training risks (V-REx) as a simpler variant. We prove that variants of REx can recover the causal mechanisms of the targets, while also providing some robustness to changes in the input distribution ("covariate shift"). By appropriately trading-off robustness to causally induced distributional shifts and covariate shift, REx is able to outperform alternative methods such as Invariant Risk Minimization in situations where these types of shift co-occur.
2021-07-19
International Conference on Machine Learning (Oral)
doi.org
proceedings.mlr.press
Preferential Temporal Difference Learning
Nishanth Anand
Doina Precup
Temporal-Difference (TD) learning is a general and very useful tool for estimating the value function of a given policy, which in turn is re… (voir plus)quired to find good policies. Generally speaking, TD learning updates states whenever they are visited. When the agent lands in a state, its value can be used to compute the TD-error, which is then propagated to other states. However, it may be interesting, when computing updates, to take into account other information than whether a state is visited or not. For example, some states might be more important than others (such as states which are frequently seen in a successful trajectory). Or, some states might have unreliable value estimates (for example, due to partial observability or lack of data), making their values less desirable as targets. We propose an approach to re-weighting states used in TD updates, both when they are the input and when they provide the target for the update. We prove that our approach converges with linear function approximation and illustrate its desirable empirical behaviour compared to other TD-style methods.
2021-07-19
International Conference on Machine Learning (Spotlight)
doi.org
proceedings.mlr.press
Randomized Exploration for Reinforcement Learning with General Value Function Approximation
Haque Ishfaq
Qiwen Cui
Viet Nguyen
Alex Ayoub
Zhuoran Yang
Zhaoran Wang
Doina Precup
Lin F. Yang
2021-07-19
International Conference on Machine Learning (Spotlight)
proceedings.mlr.press
proceedings.mlr.press
Beyond Variance Reduction: Understanding the True Impact of Baselines on Policy Optimization
Wesley Chung
Valentin Thomas
Marlos C. Machado
Nicolas Le Roux
Bandit and reinforcement learning (RL) problems can often be framed as optimization problems where the goal is to maximize average performan… (voir plus)ce while having access only to stochastic estimates of the true gradient. Traditionally, stochastic optimization theory predicts that learning dynamics are governed by the curvature of the loss function and the noise of the gradient estimates. In this paper we demonstrate that this is not the case for bandit and RL problems. To allow our analysis to be interpreted in light of multi-step MDPs, we focus on techniques derived from stochastic optimization principles (e.g., natural policy gradient and EXP3) and we show that some standard assumptions from optimization theory are violated in these problems. We present theoretical results showing that, at least for bandit problems, curvature and noise are not sufficient to explain the learning dynamics and that seemingly innocuous choices like the baseline can determine whether an algorithm converges. These theoretical findings match our empirical evaluation, which we extend to multi-state MDPs.
2021-07-17
International Conference on Machine Learning (inconnu)
doi.org
proceedings.mlr.press
Combating False Negatives in Adversarial Imitation Learning
Konrad Żołna
Chitwan Saharia
Léonard Boussioux
David Yu-tung Hui
Maxime Chevalier-Boisvert
Dzmitry Bahdanau
Yoshua Bengio
In adversarial imitation learning, a discriminator is trained to differentiate agent episodes from expert demonstrations representing the de… (voir plus)sired behavior. However, as the trained policy learns to be more successful, the negative examples (the ones produced by the agent) become increasingly similar to expert ones. Despite the fact that the task is successfully accomplished in some of the agent's trajectories, the discriminator is trained to output low values for them. We hypothesize that this inconsistent training signal for the discriminator can impede its learning, and consequently leads to worse overall performance of the agent. We show experimental evidence for this hypothesis and that the ‘False Negatives’ (i.e. successful agent episodes) significantly hinder adversarial imitation learning, which is the first contribution of this paper. Then, we propose a method to alleviate the impact of false negatives and test it on the BabyAI environment. This method consistently improves sample efficiency over the baselines by at least an order of magnitude.
2021-07-17
International Joint Conference on Neural Networks (publié)
doi.org
arxiv.org
Combating False Negatives in Adversarial Imitation Learning (Student Abstract)
Konrad Żołna
Chitwan Saharia
Léonard Boussioux
David Yu-tung Hui
Maxime Chevalier-Boisvert
Dzmitry Bahdanau
Yoshua Bengio
2021-07-17
International Joint Conference on Neural Networks (publié)
doi.org
arxiv.org
Equivariant Networks for Pixelized Spheres
Mehran Shakerinava
Siamak Ravanbakhsh
Pixelizations of Platonic solids such as the cube and icosahedron have been widely used to represent spherical data, from climate records to… (voir plus) Cosmic Microwave Background maps. Platonic solids have well-known global symmetries. Once we pixelize each face of the solid, each face also possesses its own local symmetries in the form of Euclidean isometries. One way to combine these symmetries is through a hierarchy. However, this approach does not adequately model the interplay between the two levels of symmetry transformations. We show how to model this interplay using ideas from group theory, identify the equivariant linear maps, and introduce equivariant padding that respects these symmetries. Deep networks that use these maps as their building blocks generalize gauge equivariant CNNs on pixelized spheres. These deep networks achieve state-of-the-art results on semantic segmentation for climate data and omnidirectional image processing. Code is available at https://git.io/JGiZA.
2021-07-17
International Conference on Machine Learning (inconnu)
doi.org
proceedings.mlr.press
VirtualGAN: Reducing Mode Collapse in Generative Adversarial Networks Using Virtual Mapping
Adel Abusitta
Omar Abdel Wahab
Benjamin C. M. Fung
This paper introduces a new framework for reducing mode collapse in Generative adversarial networks (GANs). The problem occurs when the gene… (voir plus)rator learns to map several various input values (z) to the same output value, which makes the generator fail to capture all modes of the true data distribution. As a result, the diversity of synthetically produced data is lower than that of the real data. To address this problem, we propose a new and simple framework for training GANs based on the concept of virtual mapping. Our framework integrates two processes into GANs: merge and split. The merge process merges multiple data points (samples) into one before training the discriminator. In this way, the generator would be trained to capture the merged-data distribution rather than the (unmerged) data distribution. After the training, the split process is applied to the generator's output in order to split its contents and produce diverse modes. The proposed framework increases the chance of capturing diverse modes through enabling an indirect or virtual mapping between an input z value and multiple data points. This, in turn, enhances the chance of generating more diverse modes. Our results show the effectiveness of our framework compared to the existing approaches in terms of reducing the mode collapse problem.
2021-07-17
IEEE International Joint Conference on Neural Network (publié)
doi.org
On generalized surrogate duality in mixed-integer nonlinear programming
Benjamin Müller
Gonzalo Muñoz
Maxime Gasse
Ambros Gleixner
Andrea Lodi
Felipe Serrano
The most important ingredient for solving mixed-integer nonlinear programs (MINLPs) to global epsilon-optimality with spatial branch and bou… (voir plus)nd is a tight, computationally tractable relaxation. Due to both theoretical and practical considerations, relaxations of MINLPs are usually required to be convex. Nonetheless, current optimization solver can often successfully handle a moderate presence of nonconvexities, which opens the door for the use of potentially tighter nonconvex relaxations. In this work, we exploit this fact and make use of a nonconvex relaxation obtained via aggregation of constraints: a surrogate relaxation. These relaxations were actively studied for linear integer programs in the 70s and 80s, but they have been scarcely considered since. We revisit these relaxations in an MINLP setting and show the computational benefits and challenges they can have. Additionally, we study a generalization of such relaxation that allows for multiple aggregations simultaneously and present the first algorithm that is capable of computing the best set of aggregations. We propose a multitude of computational enhancements for improving its practical performance and evaluate the algorithm's ability to generate strong dual bounds through extensive computational experiments.
2021-07-16
Mathematical Programming (publié)
doi.org
arxiv.org