Publications

Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment
Aamer Abdul Rahman
Pranav Agarwal
Rita Noumeir
Philippe Jouvet
Vincent Michalski
S Ebrahimi Kahou
Offline reinforcement learning has shown promise for solving tasks in safety-critical settings, such as clinical decision support. Its appli… (see more)cation, however, has been limited by the lack of interpretability and interactivity for clinicians. To address these challenges, we propose the medical decision transformer (MeDT), a novel and versatile framework based on the goal-conditioned reinforcement learning paradigm for sepsis treatment recommendation. MeDT uses the decision transformer architecture to learn a policy for drug dosage recommendation. During offline training, MeDT utilizes collected treatment trajectories to predict administered treatments for each time step, incorporating known treatment outcomes, target acuity scores, past treatment decisions, and current and past medical states. This analysis enables MeDT to capture complex dependencies among a patient's medical history, treatment decisions, outcomes, and short-term effects on stability. Our proposed conditioning uses acuity scores to address sparse reward issues and to facilitate clinician-model interactions, enhancing decision-making. Following training, MeDT can generate tailored treatment recommendations by conditioning on the desired positive outcome (survival) and user-specified short-term stability improvements. We carry out rigorous experiments on data from the MIMIC-III dataset and use off-policy evaluation to demonstrate that MeDT recommends interventions that outperform or are competitive with existing offline reinforcement learning methods while enabling a more interpretable, personalized and clinician-directed approach.
2024-07-27
ArXiv (preprint)
doi.org
arxiv.org
Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing
Dmytro Humeniuk
Foutse Khomh
Giuliano Antoniol
Evolutionary search-based techniques are commonly used for testing autonomous robotic systems. However, these approaches often rely on compu… (see more)tationally expensive simulator-based models for test scenario evaluation. To improve the computational efficiency of the search-based testing, we propose augmenting the evolutionary search (ES) with a reinforcement learning (RL) agent trained using surrogate rewards derived from domain knowledge. In our approach, known as RIGAA (Reinforcement learning Informed Genetic Algorithm for Autonomous systems testing), we first train an RL agent to learn useful constraints of the problem and then use it to produce a certain part of the initial population of the search algorithm. By incorporating an RL agent into the search process, we aim to guide the algorithm towards promising regions of the search space from the start, enabling more efficient exploration of the solution space. We evaluate RIGAA on two case studies: maze generation for an autonomous ant robot and road topology generation for an autonomous vehicle lane keeping assist system. In both case studies, RIGAA converges faster to fitter solutions and produces a better test suite (in terms of average test scenario fitness and diversity). RIGAA also outperforms the state-of-the-art tools for vehicle lane keeping assist system testing, such as AmbieGen and Frenetic.
2024-07-26
ACM Transactions on Software Engineering and Methodology (published)
doi.org
arxiv.org
On the benefits of pixel-based hierarchical policies for task generalization
T. Cristea-Platon
Bogdan Mazoure
Josh Susskind
Walter Talbott
Reinforcement learning practitioners often avoid hierarchical policies, especially in image-based observation spaces. Typically, the single-… (see more)task performance improvement over flat-policy counterparts does not justify the additional complexity associated with implementing a hierarchy. However, by introducing multiple decision-making levels, hierarchical policies can compose lower-level policies to more effectively generalize between tasks, highlighting the need for multi-task evaluations. We analyze the benefits of hierarchy through simulated multi-task robotic control experiments from pixels. Our results show that hierarchical policies trained with task conditioning can (1) increase performance on training tasks, (2) lead to improved reward and state-space generalizations in similar tasks, and (3) decrease the complexity of fine tuning required to solve novel tasks. Thus, we believe that hierarchical policies should be considered when building reinforcement learning architectures capable of generalizing between tasks.
2024-07-26
ArXiv (preprint)
doi.org
arxiv.org
Challenges in multi-task learning for fMRI-based diagnosis: Benefits for psychiatric conditions and CNVs would likely require thousands of patients
Annabelle Harvey
Clara A. Moreau
Kuldeep Kumar
Guillaume Huguet
Sebastian G.W. Urchs
Hanad Sharmarke
Khadije Jizi
Charles-Olivier Martin
Nadine Younis
Petra Tamer
Jean-Louis Martineau
Pierre Orban
Ana Isabel Silva
Jeremy Hall
Marianne B.M. van den Bree
Michael J. Owen
David E.J. Linden
Sarah Lippé
Carrie E. Bearden
Guillaume Dumas … (see 2 more)
Sébastien Jacquemont
Pierre Bellec
There is a growing interest in using machine learning (ML) models to perform automatic diagnosis of psychiatric conditions; however, general… (see more)ising the prediction of ML models to completely independent data can lead to sharp decrease in performance. Patients with different psychiatric diagnoses have traditionally been studied independently, yet there is a growing recognition of neuroimaging signatures shared across them as well as rare genetic copy number variants (CNVs). In this work, we assess the potential of multi-task learning (MTL) to improve accuracy by characterising multiple related conditions with a single model, making use of information shared across diagnostic categories and exposing the model to a larger and more diverse dataset. As a proof of concept, we first established the efficacy of MTL in a context where there is clearly information shared across tasks: the same target (age or sex) is predicted at different sites of data collection in a large functional magnetic resonance imaging (fMRI) dataset compiled from multiple studies. MTL generally led to substantial gains relative to independent prediction at each site. Performing scaling experiments on the UK Biobank, we observed that performance was highly dependent on sample size: for large sample sizes (N > 6000) sex prediction was better using MTL across three sites (N = K per site) than prediction at a single site (N = 3K), but for small samples (N 500) MTL was actually detrimental for age prediction. We then used established machine-learning methods to benchmark the diagnostic accuracy of each of the 7 CNVs (N = 19–103) and 4 psychiatric conditions (N = 44–472) independently, replicating the accuracy previously reported in the literature on psychiatric conditions. We observed that MTL hurt performance when applied across the full set of diagnoses, and complementary analyses failed to identify pairs of conditions which would benefit from MTL. Taken together, our results show that if a successful multi-task diagnostic model of psychiatric conditions were to be developed with resting-state fMRI, it would likely require datasets with thousands of patients across different diagnoses.
2024-07-25
Imaging Neuroscience (published)
doi.org
Motor Cortex Latent Dynamics Encode Spatial and Temporal Arm Movement Parameters Independently
Andrea Colins Rodriguez
Matt G. Perich
Lee E. Miller
Mark D. Humphries
The fluid movement of an arm requires multiple spatiotemporal parameters to be set independently. Recent studies have argued that arm moveme… (see more)nts are generated by the collective dynamics of neurons in motor cortex. An untested prediction of this hypothesis is that independent parameters of movement must map to independent components of the neural dynamics. Using a task where three male monkeys made a sequence of reaching movements to randomly placed targets, we show that the spatial and temporal parameters of arm movements are independently encoded in the low-dimensional trajectories of population activity in motor cortex: each movement’s direction corresponds to a fixed neural trajectory through neural state space and its speed to how quickly that trajectory is traversed. Recurrent neural network models show that this coding allows independent control over the spatial and temporal parameters of movement by separate network parameters. Our results support a key prediction of the dynamical systems view of motor cortex, and also argue that not all parameters of movement are defined by different trajectories of population activity.
2024-07-25
Journal of Neuroscience (published)
doi.org
Canada's Provincial Covid-19 Pandemic Modelling Efforts: A Review of Mathematical Models and Their Impacts on the Responses
Yiqing Xia
Jorge Luis Flores Anato
Caroline Colijin
Naveed Janjua
Michael Otterstatter
Mike Irvine
Tyler Williamson
Marie B. Varughese
Michael Li
Nathaniel Osgood
David J. D. Earn
Beate Sander
Lauren E. Cipriano
Kumar Murty
Fanyu Xiu
Arnaud Godin
David L Buckeridge
Amy Hurford
Sharmistha Mishra
Mathieu Maheu-Giroux
2024-07-24
Canadian journal of public health (published)
doi.org
Intrinsic structural covariation links cerebellum subregions to the cerebral cortex
Zilong Wang
Jörn Diedrichsen
Karin Saltoun
Christopher Steele
Sheeba Rani Arnold-Anteraper
B.T. Thomas Yeo
Jeremy Schmahmann
Danilo Bzdok
The human cerebellum is increasingly recognized to be involved in non-motor and higher-order cognitive functions. Yet, its ties with the ent… (see more)ire cerebral cortex have not been holistically studied in a whole-brain exploration with a unified analytical framework. Here, we characterized disso-ciable cortical-cerebellar structural covariation patterns across the brain in n=38,527 UK Bio-bank participants. Our results invigorate previous observations in that important shares of corti-cal-cerebellar structural covariation are described as i) a dissociation between the higher-level cognitive system and lower-level sensorimotor system, as well as ii) an anticorrelation between the visual-attention system and advanced associative networks within the cerebellum. We also discovered a novel pattern of ipsilateral, rather than contralateral, cerebral-cerebellar associations. Furthermore, phenome-wide association assays revealed key phenotypes, including cognitive phenotypes, lifestyle, physical properties, and blood assays, associated with each decomposed covariation pattern, helping to understand their real-world implications. This systems neurosci-ence view paves the way for future studies to explore the implications of these structural covaria-tions, potentially illuminating new pathways in our understanding of neurological and cognitive disorders.
2024-07-24
Journal of Neurophysiology (published)
doi.org
Multi-Fidelity Active Learning with GFlowNets
Alex Hernández-García
Nikita Saxena
Moksh J. Jain
Cheng-Hao Liu
Yoshua Bengio
In the last decades, the capacity to generate large amounts of data in science and engineering applications has been growing steadily. Meanw… (see more)hile, the progress in machine learning has turned it into a suitable tool to process and utilise the available data. Nonetheless, many relevant scientific and engineering problems present challenges where current machine learning methods cannot yet efficiently leverage the available data and resources. For example, in scientific discovery, we are often faced with the problem of exploring very large, high-dimensional spaces, where querying a high fidelity, black-box objective function is very expensive. Progress in machine learning methods that can efficiently tackle such problems would help accelerate currently crucial areas such as drug and materials discovery. In this paper, we propose the use of GFlowNets for multi-fidelity active learning, where multiple approximations of the black-box function are available at lower fidelity and cost. GFlowNets are recently proposed methods for amortised probabilistic inference that have proven efficient for exploring large, high-dimensional spaces and can hence be practical in the multi-fidelity setting too. Here, we describe our algorithm for multi-fidelity active learning with GFlowNets and evaluate its performance in both well-studied synthetic tasks and practically relevant applications of molecular discovery. Our results show that multi-fidelity active learning with GFlowNets can efficiently leverage the availability of multiple oracles with different costs and fidelities to accelerate scientific discovery and engineering design.
2024-07-24
TMLR (accepted)
doi.org
openreview.net
Structural covariation between cerebellum and neocortex intrinsic structural covariation links cerebellum subregions to the cerebral cortex
Zilong Wang
Jörn Diedrichsen
Karin Saltoun
Christopher Steele
Sheeba Rani Arnold-Anteraper
B. T. Thomas Yeo
Jeremy D. Schmahmann
Danilo Bzdok
Cerebellum’s association with the entire cerebral cortex has not been holistically studied in a unified way. Here, we conjointly character… (see more)ize the population-level cortical-cerebellar structural covariation patterns leveraging ∼40,000 UK Biobank participants whole brain structural scans and ∼1,000 phenotypes. We revitalize the previous hypothesis of an anticorrelation between the visual-attention system and advanced associative networks within the cerebellum. We also discovered a novel ipsilateral cerebral-cerebellar associations. Phenome-wide association (PheWAS) revealed real-world implications of the structural covariation patterns.
2024-07-24
Journal of Neurophysiology (unknown)
doi.org
Development of Error Passing Network for Optimizing the Prediction of VO$_2$ peak in Childhood Acute Leukemia Survivors
Nicolas Raymond
Hakima Laribi
Maxime Caru
Mehdi Mitiche
Valerie Marcil
Maja Krajinovic
Daniel Curnier
Daniel Sinnett
Martin Vallières
Approximately two-thirds of survivors of childhood acute lymphoblastic leukemia (ALL) cancer develop late adverse effects post-treatment. Pr… (see more)ior studies explored prediction models for personalized follow-up, but none integrated the usage of neural networks to date. In this work, we propose the Error Passing Network (EPN), a graph-based method that leverages relationships between samples to propagate residuals and adjust predictions of any machine learning model. We tested our approach to estimate patients’ \vo peak, a reliable indicator of their cardiac health. We used the EPN in conjunction with several baseline models and observed up to 12.16% improvement in the mean average percentage error compared to the last established equation predicting \vo peak in childhood ALL survivors. Along with this performance improvement, our final model is more efficient considering that it relies only on clinical variables that can be self-reported by patients, therefore removing the previous need of executing a resource-consuming physical test.
2024-07-23
Proceedings of the fifth Conference on Health, Inference, and Learning (published)
proceedings.mlr.press
proceedings.mlr.press
Adaptive Accompaniment with ReaLchords
Yusong Wu
Tim Cooijmans
Kyle Kastner
Adam Roberts
Ian Simon
Alexander Scarlatos
Chris Donahue
Cassie Tarakajian
Shayegan Omidshafiei
Aaron Courville
Pablo Samuel Castro
Natasha Jaques
Cheng-Zhi Anna Huang
Jamming requires coordination, anticipation, and collaborative creativity between musicians. Current generative models of music produce expr… (see more)essive output but are not able to generate in an \emph{online} manner, meaning simultaneously with other musicians (human or otherwise). We propose ReaLchords, an online generative model for improvising chord accompaniment to user melody. We start with an online model pretrained by maximum likelihood, and use reinforcement learning to finetune the model for online use. The finetuning objective leverages both a novel reward model that provides feedback on both harmonic and temporal coherency between melody and chord, and a divergence term that implements a novel type of distillation from a teacher model that can see the future melody. Through quantitative experiments and listening tests, we demonstrate that the resulting model adapts well to unfamiliar input and produce fitting accompaniment. ReaLchords opens the door to live jamming, as well as simultaneous co-creation in other modalities.
2024-07-22
International Conference on Machine Learning (Accept (Poster))
doi.org
proceedings.mlr.press
Code as Reward: Empowering Reinforcement Learning with VLMs
David Venuto
Sami Nur Islam
Martin Klissarov
Doina Precup
Sherry Yang
Ankit Anand
Pre-trained Vision-Language Models (VLMs) are able to understand visual concepts, describe and decompose complex tasks into sub-tasks, and p… (see more)rovide feedback on task completion. In this paper, we aim to leverage these capabilities to support the training of reinforcement learning (RL) agents. In principle, VLMs are well suited for this purpose, as they can naturally analyze image-based observations and provide feedback (reward) on learning progress. However, inference in VLMs is computationally expensive, so querying them frequently to compute rewards would significantly slowdown the training of an RL agent. To address this challenge, we propose a framework named Code as Reward (VLM-CaR). VLM-CaR produces dense reward functions from VLMs through code generation, thereby significantly reducing the computational burden of querying the VLM directly. We show that the dense rewards generated through our approach are very accurate across a diverse set of discrete and continuous environments, and can be more effective in training RL policies than the original sparse environment rewards.
2024-07-22
International Conference on Machine Learning (Accept (Spotlight))
doi.org
proceedings.mlr.press