Publications

Learning Modular Safe Policies in the Bandit Setting with Application to Adaptive Clinical Trials
Hossein Aboutalebi
Doina Precup
Tibor Schuster
The stochastic multi-armed bandit problem is a well-known model for studying the exploration-exploitation trade-off. It has significant poss… (see more)ible applications in adaptive clinical trials, which allow for dynamic changes in the treatment allocation probabilities of patients. However, most bandit learning algorithms are designed with the goal of minimizing the expected regret. While this approach is useful in many areas, in clinical trials, it can be sensitive to outlier data, especially when the sample size is small. In this paper, we define and study a new robustness criterion for bandit problems. Specifically, we consider optimizing a function of the distribution of returns as a regret measure. This provides practitioners more flexibility to define an appropriate regret measure. The learning algorithm we propose to solve this type of problem is a modification of the BESA algorithm [Baransi et al., 2014], which considers a more general version of regret. We present a regret bound for our approach and evaluate it empirically both on synthetic problems as well as on a dataset from the clinical trial literature. Our approach compares favorably to a suite of standard bandit algorithms.
2019-03-03
ArXiv (preprint)
arxiv.org
arxiv.org
Deep Learning for Automated Segmentation of Liver Lesions at CT in Patients with Colorectal Cancer Liver Metastases.
Eugene Vorontsov
Milena Cerny
Philippe Régnier
Lisa Di Jorio
Christopher Pal
Réal Lapointe
Franck Vandenbroucke-Menu
Simon Turcotte
Samuel Kadoury
An Tang
Purpose To evaluate the performance, agreement, and efficiency of a fully convolutional network (FCN) for liver lesion detection and segment… (see more)ation at CT examinations in patients with colorectal liver metastases (CLMs). Materials and Methods This retrospective study evaluated an automated method using an FCN that was trained, validated, and tested with 115, 15, and 26 contrast material-enhanced CT examinations containing 261, 22, and 105 lesions, respectively. Manual detection and segmentation by a radiologist was the reference standard. Performance of fully automated and user-corrected segmentations was compared with that of manual segmentations. The interuser agreement and interaction time of manual and user-corrected segmentations were assessed. Analyses included sensitivity and positive predictive value of detection, segmentation accuracy, Cohen κ, Bland-Altman analyses, and analysis of variance. Results In the test cohort, for lesion size smaller than 10 mm (n = 30), 10-20 mm (n = 35), and larger than 20 mm (n = 40), the detection sensitivity of the automated method was 10%, 71%, and 85%; positive predictive value was 25%, 83%, and 94%; Dice similarity coefficient was 0.14, 0.53, and 0.68; maximum symmetric surface distance was 5.2, 6.0, and 10.4 mm; and average symmetric surface distance was 2.7, 1.7, and 2.8 mm, respectively. For manual and user-corrected segmentation, κ values were 0.42 (95% confidence interval: 0.24, 0.63) and 0.52 (95% confidence interval: 0.36, 0.72); normalized interreader agreement for lesion volume was -0.10 ± 0.07 (95% confidence interval) and -0.10 ± 0.08; and mean interaction time was 7.7 minutes ± 2.4 (standard deviation) and 4.8 minutes ± 2.1 (P .001), respectively. Conclusion Automated detection and segmentation of CLM by using deep learning with convolutional neural networks, when manually corrected, improved efficiency but did not substantially change agreement on volumetric measurements.© RSNA, 2019Supplemental material is available for this article.
2019-02-28
Radiology: Artificial Intelligence (published)
doi.org
Reinforcement Learning in Stationary Mean-field Games
Jayakumar Subramanian
Aditya Mahajan
Multi-agent reinforcement learning has made significant progress in recent years, but it remains a hard problem. Hence, one often resorts to… (see more) developing learning algorithms for specific classes of multi-agent systems. In this paper we study reinforcement learning in a specific class of multi-agent systems systems called mean-field games. In particular, we consider learning in stationary mean-field games. We identify two different solution concepts---stationary mean-field equilibrium and stationary mean-field social-welfare optimal policy---for such games based on whether the agents are non-cooperative or cooperative, respectively. We then generalize these solution concepts to their local variants using bounded rationality based arguments. For these two local solution concepts, we present two reinforcement learning algorithms. We show that the algorithms converge to the right solution under mild technical conditions and demonstrate this using two numerical examples.
2019-02-28
Adaptive Agents and Multi-Agent Systems (published)
doi.org
Stochastic Bit-Wise Iterative Decoding of Polar Codes
Kaining Han
Junchao Wang
Warren J. Gross
Jianhao Hu
Polar codes have received recent attention due to their potential to be applied in advanced wireless communication protocols such as the fif… (see more)th generation mobile communication system (5G). Among the existing decoding algorithms, Belief Propagation (BP) exhibits high-throughput, low-latency, and soft output with a high hardware cost. Stochastic computing, as a form of approximate computing, provides a potential low-cost implementation solution for the BP algorithm. However, existing stochastic BP decoders suffer from a relatively long decoding latency resulting in low hardware efficiency. In this paper, a novel bit-wise iterative stochastic decoding architecture for the BP algorithm is proposed to improve the throughput and hardware efficiency. By utilizing the frozen bits of polar codes and stochastic computing, multiple novel optimization methods are presented to further speed up convergence and increase the hardware efficiency.
2019-02-28
IEEE Transactions on Signal Processing (published)
doi.org
Prediction of Progression in Multiple Sclerosis Patients
Adrian Tousignant
Paul Lemaitre
Doina Precup
Douglas Arnold
Tal Arbel
We present the first automatic end-to-end deep learning framework for the prediction of future patient disability progression (one year from… (see more) baseline) based on multi-modal brain Magnetic Resonance Images (MRI) of patients with Multiple Sclerosis (MS). The model uses parallel convolutional pathways, an idea introduced by the popular Inception net and is trained and tested on two large proprietary, multi-scanner, multi-center, clinical trial datasets of patients with Relapsing-Remitting Multiple Sclerosis (RRMS). Experiments on 465 patients on the placebo arms of the trials indicate that the model can accurately predict future disease progression, measured by a sustained increase in the extended disability status scale (EDSS) score over time. Using only the multi-modal MRI provided at baseline, the model achieves an AUC of 0.66 +- 0.055. However, when supplemental lesion label masks are provided as inputs as well, the AUC increases to 0.701 +- 0.027. Furthermore, we demonstrate that uncertainty estimates based on Monte Carlo dropout sample variance correlate with errors made by the model. Clinicians provided with the predictions computed by the model can therefore use the associated uncertainty estimates to assess which scans require further examination.
2019-02-27
MIDL.io/2019/Conference (poster)
openreview.net
openreview.net
The Termination Critic
Anna Harutyunyan
Will Dabney
Diana Borsa
Nicolas Heess
Remi Munos
Doina Precup
In this work, we consider the problem of autonomously discovering behavioral abstractions, or options, for reinforcement learning agents. We… (see more) propose an algorithm that focuses on the termination function, as opposed to - as is common - the policy. The termination function is usually trained to optimize a control objective: an option ought to terminate if another has better value. We offer a different, information-theoretic perspective, and propose that terminations should focus instead on the compressibility of the option’s encoding - arguably a key reason for using abstractions. To achieve this algorithmically, we leverage the classical options framework, and learn the option transition model as a "critic" for the termination function. Using this model, we derive gradients that optimize the desired criteria. We show that the resulting options are non-trivial, intuitively meaningful, and useful for learning.
2019-02-25
ArXiv (preprint)
proceedings.mlr.press
proceedings.mlr.press
The Termination Critic
Anna Harutyunyan
Will Dabney
Diana Borsa
Nicolas Heess
Remi Munos
Doina Precup
In this work, we consider the problem of autonomously discovering behavioral abstractions, or options, for reinforcement learning agents. We… (see more) propose an algorithm that focuses on the termination function, as opposed to - as is common - the policy. The termination function is usually trained to optimize a control objective: an option ought to terminate if another has better value. We offer a different, information-theoretic perspective, and propose that terminations should focus instead on the compressibility of the option’s encoding - arguably a key reason for using abstractions.To achieve this algorithmically, we leverage the classical options framework, and learn the option transition model as a “critic” for the termination function. Using this model, we derive gradients that optimize the desired criteria. We show that the resulting options are non-trivial, intuitively meaningful, and useful for learning.
2019-02-25
ArXiv (preprint)
arxiv.org
arxiv.org
Hyperbolic Discounting and Learning over Multiple Horizons
William Fedus
Carles Gelada
Yoshua Bengio
Bellemare Marc-Emmanuel
Hugo Larochelle
Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future re… (see more)wards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences. In this work we revisit the fundamentals of discounting in RL and bridge this disconnect by implementing an RL agent that acts via hyperbolic discounting. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over a strong value-based RL agent, Rainbow.
2019-02-18
ArXiv (preprint)
openreview.net
openreview.net
Predicting conversion to psychosis in clinical high risk patients using resting-state functional MRI features
Jolie Mcdonnell
W. Hord
Jenna Reinen
Pablo Polosecki
Irina Rish
Guillermo Cecchi
Recent progress in artificial intelligence provides researchers with a powerful set of machine learning tools for analyzing brain imaging da… (see more)ta. In this work, we explore a variety of classification algorithms and functional network features derived from resting-state fMRI data collected from clinical high-risk (prodromal schizophrenia) patients and controls, trying to identify features predictive of conversion to psychosis among a subset of CHR patients. While there are many existing studies suggesting that functional network features can be highly discriminative of schizophrenia when analyzing fMRI of patients suffering from the disease vs controls, few studies attempt to explore a similar approach to actual prediction of future psychosis development ahead of time, in the prodromal stage. Our preliminary results demonstrate the potential of fMRI functional network features to predict the conversion to psychosis in CHR patients. However, given the high variance of our results across different classifiers and subsets of data, a more extensive empirical investigation is required to reach more robust conclusions.
2019-02-15
Medical Imaging 2019: Biomedical Applications in Molecular, Structural, and Functional Imaging (published)
doi.org
Anytime Tail Averaging
Nicolas Roux
Tail averaging consists in averaging the last examples in a stream. Common techniques either have a memory requirement which grows with the … (see more)number of samples to average, are not available at every timestep or do not accomodate growing windows. We propose two techniques with a low constant memory cost that perform tail averaging with access to the average at every time step. We also show how one can improve the accuracy of that average at the cost of increased memory consumption.
2019-02-12
ArXiv (preprint)
arxiv.org
arxiv.org
Dendritic solutions to the credit assignment problem
Blake Aaron Richards
Timothy P Lillicrap
2019-01-31
Current Opinion in Neurobiology (published)
doi.org
Equivalence of Equilibrium Propagation and Recurrent Backpropagation
Benjamin Scellier
Yoshua Bengio
Recurrent backpropagation and equilibrium propagation are supervised learning algorithms for fixed-point recurrent neural networks, which di… (see more)ffer in their second phase. In the first phase, both algorithms converge to a fixed point that corresponds to the configuration where the prediction is made. In the second phase, equilibrium propagation relaxes to another nearby fixed point corresponding to smaller prediction error, whereas recurrent backpropagation uses a side network to compute error derivatives iteratively. In this work, we establish a close connection between these two algorithms. We show that at every moment in the second phase, the temporal derivatives of the neural activities in equilibrium propagation are equal to the error derivatives computed iteratively by recurrent backpropagation in the side network. This work shows that it is not required to have a side network for the computation of error derivatives and supports the hypothesis that in biological neural networks, temporal derivatives of neural activities may code for error signals.
2019-01-31
Neural Computation (published)
doi.org
openreview.net