Publications

Universal Successor Features for Transfer Reinforcement Learning

Chen Ma

Dylan R. Ashley

Junfeng Wen

Yoshua Bengio

Transfer in Reinforcement Learning (RL) refers to the idea of applying knowledge gained from previous tasks to solving related tasks. Learni… (see more)ng a universal value function (Schaul et al., 2015), which generalizes over goals and states, has previously been shown to be useful for transfer. However, successor features are believed to be more suitable than values for transfer (Dayan, 1993; Barreto et al.,2017), even though they cannot directly generalize to new goals. In this paper, we propose (1) Universal Successor Features (USFs) to capture the underlying dynamics of the environment while allowing generalization to unseen goals and (2) a flexible end-to-end model of USFs that can be trained by interacting with the environment. We show that learning USFs is compatible with any RL algorithm that learns state values using a temporal difference method. Our experiments in a simple gridworld and with two MuJoCo environments show that USFs can greatly accelerate training when learning multiple tasks and can effectively transfer knowledge to new tasks.

2018-09-26

ArXiv (preprint)

Unsupervised one-to-many image translation

Samuel Lavoie-Marchildon

R Devon Hjelm

2018-09-26

(published)

W2GAN: RECOVERING AN OPTIMAL TRANSPORT MAP WITH A GAN

Leygonie Jacob*

Jennifer She*

Amjad Almahairi

Sai Rajeswar

Aaron Courville

2018-09-26

(published)

Where Off-Policy Deep Reinforcement Learning Fails

Scott Fujimoto

David Meger

Doina Precup

This work examines batch reinforcement learning–the task of maximally exploiting a given batch of off-policy data, without further data co… (see more)llection. We demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are only capable of learning with data correlated to their current policy, making them ineffective for most off-policy applications. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space to force the agent towards behaving on-policy with respect to a subset of the given data. We extend this notion to deep reinforcement learning, and to the best of our knowledge, present the first continuous control deep reinforcement learning algorithm which can learn effectively from uncorrelated off-policy data.

2018-09-26

(published)

Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio

Stanisław Jastrzębski

Amos Storkey

2018-09-26

Artificial Neural Networks and Machine Learning – ICANN 2018 (published)

Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation

Tanya Nair

Doina Precup

Douglas L. Arnold

Deep learning (DL) networks have recently been shown to outperform other segmentation methods on various public, medical-image challenge dat… (see more)asets [3,11,16], especially for large pathologies. However, in the context of diseases such as Multiple Sclerosis (MS), monitoring all the focal lesions visible on MRI sequences, even very small ones, is essential for disease staging, prognosis, and evaluating treatment efficacy. Moreover, producing deterministic outputs hinders DL adoption into clinical routines. Uncertainty estimates for the predictions would permit subsequent revision by clinicians. We present the first exploration of multiple uncertainty estimates based on Monte Carlo (MC) dropout [4] in the context of deep networks for lesion detection and segmentation in medical images. Specifically, we develop a 3D MS lesion segmentation CNN, augmented to provide four different voxel-based uncertainty measures based on MC dropout. We train the network on a proprietary, large-scale, multi-site, multi-scanner, clinical MS dataset, and compute lesion-wise uncertainties by accumulating evidence from voxel-wise uncertainties within detected lesions. We analyze the performance of voxel-based segmentation and lesion-level detection by choosing operating points based on the uncertainty. Empirical evidence suggests that uncertainty measures consistently allow us to choose superior operating points compared only using the network's sigmoid output as a probability.

2018-09-25

Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (published)

Nazanin Mohammadi Sepahvand

arxiv.org

How can deep learning advance computational modeling of sensory information processing?

Jessica A.F. Thompson

Yoshua Bengio

Elia Formisano

Marc Schönwiesner

Deep learning, computational neuroscience, and cognitive science have overlapping goals related to understanding intelligence such that perc… (see more)eption and behaviour can be simulated in computational systems. In neuroimaging, machine learning methods have been used to test computational models of sensory information processing. Recently, these model comparison techniques have been used to evaluate deep neural networks (DNNs) as models of sensory information processing. However, the interpretation of such model evaluations is muddied by imprecise statistical conclusions. Here, we make explicit the types of conclusions that can be drawn from these existing model comparison techniques and how these conclusions change when the model in question is a DNN. We discuss how DNNs are amenable to new model comparison techniques that allow for stronger conclusions to be made about the computational mechanisms underlying sensory information processing.

2018-09-24

ArXiv (preprint)

arxiv.org

On the Learning Dynamics of Deep Neural Networks

Remi Tachet des Combes

Mohammad Pezeshki

Samira Shabanian

Aaron Courville

Yoshua Bengio

While a lot of progress has been made in recent years, the dynamics of learning in deep nonlinear neural networks remain to this day largely… (see more) misunderstood. In this work, we study the case of binary classification and prove various properties of learning in such networks under strong assumptions such as linear separability of the data. Extending existing results from the linear case, we confirm empirical observations by proving that the classification error also follows a sigmoidal shape in nonlinear architectures. We show that given proper initialization, learning expounds parallel independent modes and that certain regions of parameter space might lead to failed training. We also demonstrate that input norm and features' frequency in the dataset lead to distinct convergence speeds which might shed some light on the generalization capabilities of deep neural networks. We provide a comparison between the dynamics of learning with cross-entropy and hinge losses, which could prove useful to understand recent progress in the training of generative adversarial networks. Finally, we identify a phenomenon that we baptize \textit{gradient starvation} where the most frequent features in a dataset prevent the learning of other less frequent but equally informative features.

2018-09-17

ArXiv (preprint)

arxiv.org

CNN Prediction of Future Disease Activity for Multiple Sclerosis Patients from Baseline MRI and Lesion Labels

Tal Hassner

Douglas Arnold

2018-09-15

BrainLes@MICCAI (published)

3D U-Net for Brain Tumour Segmentation

Raghav Mehta

2018-09-15

BrainLes@MICCAI (published)

How to Exploit Weaknesses in Biomedical Challenge Design and Organization

Annika Reinke

Matthias Eisenmann

Sinan Onogur

Marko Stankovic

Patrick Scholz

Peter M. Full

Hrvoje Bogunovic

Bennett Landman

Oskar Maier

Bjoern Menze

Gregory C. Sharp

Korsuk Sirinukunwattana

Stefanie Speidel

F. V. D. Sommen

Guoyan Zheng

Henning Müller

Michal Kozubek

Andrew P. Bradley

Pierre Jannin … (see 2 more)

Annette Kopp-Schneider

Lena Maier-Hein

2018-09-12

Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (published)