Improving medical image analysis with AI

Using deep learning to predict individual outcome and response to treatment for patients with neurological diseases.

A head x-ray taken by a computer-assisted tomographic (CAT) scanner.


Personalized medicine tailors treatment to the individual patient, and can be particularly crucial in chronic diseases such as multiple sclerosis (MS), where timely and effective treatment is essential to prevent morbidity and reduce disability.


While current decisions are based on general statistics and population-based clinical markers, harnessing patient images for personalized predictions can enable individualized treatment recommendations that better account for variability in drug efficacy across patients. 


Yet these models remain underexplored, and the open challenges presented by real-world clinical settings hamper their safe deployment.


We develop deep learning models leveraging medical images for real-world clinical contexts and improved patient care. This could result in tools to assist clinicians with the selection of optimal treatments for individual patients with chronic, incurable disease, as well as improved clinical trial analysis. 


We seek to develop modern deep learning frameworks which accurately predict future patient outcomes and individual treatment effects, requiring them to overcome challenges presented by real-world data. 


We also need to ensure the safety and reliability of the models for successful clinical deployment. 

We have developed the first causal inference model for personalized medicine, based on 3D magnetic resonance images (MRI) acquired from patients suffering from neurological diseases during clinical trials.


In particular, this model can accurately predict the future worsening of a patient's disease under the effect of all possible therapies (and a placebo). The aim was also to identify individuals likely to respond to different treatments in heterogeneous populations, by predicting future individual treatment effects. At the individual level, this can be used to improve treatment recommendations: the framework helped identify sub-populations of patients who responded to treatments.


We are striving to improve model safety and reliability by building the first uncertainty-aware causal models for imaging-based personalized medicine. Communicating the probability of response to treatment would enable more informed recommendations to be made.


We tested the model on a large clinical trial dataset of multiple sclerosis patients, provided by the International Progressive MS Alliance.

Featured Content

Tal Arbel: Approaching Multiple Sclerosis with Machine Learning

NeurologyLive spoke to Tal Arbel, Mila Associate Academic Member, at the ACTRIMS forum to find out more about the work being done in this field.

The Promise of AI for Personalized Medicine based on Medical Images

Joshua Durso-Finley (McGill/Mila) speaks at the USS (UNIQUE Student Symposium) in a session on NeuroAI in medical research.

Photo of Tal Arbel

I'm proud to be developing AI tools to improve outcomes for patients with long-term incurable diseases, together with my incredible research team.

Tal Arbel, Professor, McGill University, Core Academic Member, Mila


Patients analyzed

Over 1,800 MS patients have been analyzed as part of the project.


Treatments evaluated

Experiments were conducted on 5 different treatments. 


Randomized clinical trials

The model was tested in 4 randomized clinical trials.


Publications Directory: Probabilistic Vision Group (PVG)
The PVG is an interdisciplinary research lab dedicated to the development of probabilistic machine learning frameworks in computer vision for a wide range of real-world applications in neurology and neurosurgery.
Improving Image-Based Precision Medicine with Uncertainty-Aware Causal Models
Interview with PhD student Joshua Durso-Finley about his paper and his research for MICCAI 2023.


Improving Image-Based Precision Medicine with Uncertainty-Aware Causal Models
Joshua D. Durso-Finley
Jean-Pierre R. Falet
Raghav Mehta
Douglas Arnold
Nick Pawlowski
Image-based precision medicine aims to personalize treatment decisions based on an individual's unique imaging features so as to improve the… (see more)ir clinical outcome. Machine learning frameworks that integrate uncertainty estimation as part of their treatment recommendations would be safer and more reliable. However, little work has been done in adapting uncertainty estimation techniques and validation metrics for precision medicine. In this paper, we use Bayesian deep learning for estimating the posterior distribution over factual and counterfactual outcomes on several treatments. This allows for estimating the uncertainty for each treatment option and for the individual treatment effects (ITE) between any two treatments. We train and evaluate this model to predict future new and enlarging T2 lesion counts on a large, multi-center dataset of MR brain images of patients with multiple sclerosis, exposed to several treatments during randomized controlled trials. We evaluate the correlation of the uncertainty estimate with the factual error, and, given the lack of ground truth counterfactual outcomes, demonstrate how uncertainty for the ITE prediction relates to bounds on the ITE error. Lastly, we demonstrate how knowledge of uncertainty could modify clinical decision-making to improve individual patient and clinical trial outcomes.
Personalized Prediction of Future Lesion Activity and Treatment Effect in Multiple Sclerosis from Baseline MRI
Joshua D. Durso-Finley
Jean-Pierre R. Falet
Brennan Nichyporuk
Douglas Arnold
Precision medicine for chronic diseases such as multiple sclerosis (MS) involves choosing a treatment which best balances efficacy and side … (see more)effects/preferences for individual patients. Making this choice as early as possible is important, as delays in finding an effective therapy can lead to irreversible disability accrual. To this end, we present the first deep neural network model for individualized treatment decisions from baseline magnetic resonance imaging (MRI) (with clinical information if available) for MS patients which (a) predicts future new and enlarging T2 weighted (NE-T2) lesion counts on follow-up MRI on multiple treatments and (b) estimates the conditional average treatment effect (CATE), as defined by the predicted future suppression of NE-T2 lesions, between different treatment options relative to placebo. Our model is validated on a proprietary federated dataset of 1817 multi-sequence MRIs acquired from MS patients during four multi-centre randomized clinical trials. Our framework achieves high average precision in the binarized regression of future NE-T2 lesions on five different treatments, identifies heterogeneous treatment effects, and provides a personalized treatment recommendation that accounts for treatment-associated risk (side effects, patient preference, administration difficulties).
Estimating individual treatment effect on disability progression in multiple sclerosis using deep learning
Jean-Pierre R. Falet
Joshua D. Durso-Finley
Brennan Nichyporuk
Julien Schroeter
Francesca Bovis
Maria-Pia Sormani
Douglas Arnold

Meet the Team

Mila Members
Core Academic Member
Portrait of Tal Arbel
Full Professor, McGill University, Department of Electrical and Computer Engineering
Canada CIFAR AI Chair
Core Academic Member
Portrait of Doina Precup
Associate Professor, McGill University, School of Computer Science
Canada CIFAR AI Chair
Portrait of Brennan Nichyporuk is unavailable
Research Scientist, Innovation, Development and Technologies
Portrait of Berardino Barile is unavailable
Postdoctorate - McGill University
Portrait of Jean-pierre Falet is unavailable
PhD - Université de Montréal
Portrait of Joshua Durso-finley is unavailable
PhD - McGill University
Other Members
Raghav Mehta (Mila Alumni)
Julien Schroeter (Mila Alumni)
Nick Pawlowski (Microsoft Research)
Dr. D.L. Arnold (McGill University, Montreal Neurological Institute, NeuroRx Research)