Irina Rish

andrew.williams@mila.quebec

Amin Darabi

PhD - Université de Montréal

amin.darabi@mila.quebec

Amin Memarian

Independent visiting researcher

memariaa@mila.quebec

Andrei Mircea Romascanu

PhD - Université de Montréal

PhD - Université de Montréal

arian.khorasani@mila.quebec

Arian Khorasani

Master's Research - Université de Montréal

arnav-kumar.jain@mila.quebec

Arjun Ashok

PhD

Co-supervisor :

Alexandre Drouin

arjun.ashok@mila.quebec

PhD - Université de Montréal

PhD - Université de Montréal

Co-supervisor :

Collaborating researcher

ayush.kaushal@mila.quebec

Benjamin Therien

PhD - Université de Montréal

Co-supervisor :

benjamin.therien@mila.quebec

Postdoctorate - Université de Montréal

connor.brennan@mila.quebec

Daria Yasafova

Research Intern - Technical University of Munich

daria.yasafova@mila.quebec

Dave Whipps

Master's Research - Université de Montréal

whippsda@mila.quebec

diganta.misra@mila.quebec

Diganta Misra

Master's Research - Université de Montréal

Postdoctorate

Principal supervisor :

Nicolas Le Roux

ekaterina.lobacheva@mila.quebec

PhD - McGill University

Principal supervisor :

Blake Richards

ethan.caballero@mila.quebec

george.adamopoulos@mila.quebec

George Adamopoulos

Research Intern

gopeshh.subbaraj@mila.quebec

Germán Abrevaya

Independent visiting researcher - Université de Montréal

Co-supervisor :

PhD - Université de Montréal

Gwen Legate

PhD - Concordia University

Principal supervisor :

gwendolyne.legate@mila.quebec

Ivan Anokhin

PhD - Université de Montréal

Co-supervisor :

Samira Ebrahimi Kahou

ivan.anokhin@mila.quebec

Sarath Chandar Anbil Parthipan

Kshitij Gupta

Collaborating Alumni - Université de Montréal

Co-supervisor :

kshitij.gupta@mila.quebec

Mahta Ramezanian

Master's Research - Université de Montréal

Co-supervisor :

mahta.ramezanian@mila.quebec

Matthew Riemer

PhD - Université de Montréal

matthew.riemer@mila.quebec

Maximilian Puelma Touzel

Collaborating researcher

PhD - Université de Montréal

arefinmr@mila.quebec

Mohammad Pezeshki

Collaborating researcher

pezeshki@mila.quebec

Mohammad Javad Darvishi Bayazi

PhD - Université de Montréal

mohammad-javad.darvishi-bayasi@mila.quebec

Mojtaba Faramarzi

PhD - Université de Montréal

faramarm@mila.quebec

Motahareh Pourrahimi

PhD - McGill University

Principal supervisor :

Pouya Bashivan

motahareh.pourrahimi@mila.quebec

nadhir.hassen@mila.quebec

Nadhir Hassen

Research Intern - Université de Montréal

Neeraj Kumar

Professional Master's - Université de Montréal

neeraj.kumar@mila.quebec

Nizar Islah

PhD - Université de Montréal

Principal supervisor :

Eilif Benjamin Muller

nizar.islah@mila.quebec

paolo.cudrano@mila.quebec

Omar Younis

Research Intern - Université de Montréal

omar.younis@mila.quebec

Collaborating researcher - Politecnico di Milano

Pascal Tikeng Notsawo

PhD - Université de Montréal

Co-supervisor :

pascal.tikeng@mila.quebec

Collaborating researcher

prateek.humane@mila.quebec

Postdoctorate - Université de Montréal

Principal supervisor :

Sarath Chandar Anbil Parthipan

quentin.fournier@mila.quebec

Remus Mocanu

Master's Research - Université de Montréal

remus.mocanu@mila.quebec

Reza Bayat

Master's Research - Université de Montréal

Co-supervisor :

Pouya Bashivan

reza.bayat@mila.quebec

rishika.bhagwatkar@mila.quebec

Rishika Bhagwatkar

Master's Research - Université de Montréal

Collaborating researcher - Université de Montréal

roland.riachi@mila.quebec

Amin Sayed Mohammadamin Mansouri Tehrani

Master's Research - Université de Montréal

sayed.mansouri-tehrani@mila.quebec

Master's Research - Université de Montréal

sparsha.mishra@mila.quebec

Tejas Tejas

Master's Research - Université de Montréal

tejas.vaidhya@mila.quebec

PhD - Université de Montréal

Co-supervisor :

Eilif Benjamin Muller

timothy.nest@mila.quebec

Vaibhav Singh

PhD - Concordia University

Principal supervisor :

vaibhav.singh@mila.quebec

Zahra Sheikhbahaee

Postdoctorate - Université de Montréal

Principal supervisor :

zahra.sheikhbahaee@mila.quebec

Publications

Deep Generative Sampling in the Dual Divergence Space: A Data-efficient&Interpretative Approach for Generative AI

Sahil Garg

Anderson Schneider

Anant Raj

Kashif Rasul

Yuriy Nevmyvaka

S. Gopal

Amit Dhurandhar

Guillermo A. Cecchi

Building on the remarkable achievements in generative sampling of natural images, we propose an innovative challenge, potentially overly amb… (see more)itious, which involves generating samples of entire multivariate time series that resemble images. However, the statistical challenge lies in the small sample size, sometimes consisting of a few hundred subjects. This issue is especially problematic for deep generative models that follow the conventional approach of generating samples from a canonical distribution and then decoding or denoising them to match the true data distribution. In contrast, our method is grounded in information theory and aims to implicitly characterize the distribution of images, particularly the (global and local) dependency structure between pixels. We achieve this by empirically estimating its KL-divergence in the dual form with respect to the respective marginal distribution. This enables us to perform generative sampling directly in the optimized 1-D dual divergence space. Specifically, in the dual space, training samples representing the data distribution are embedded in the form of various clusters between two end points. In theory, any sample embedded between those two end points is in-distribution w.r.t. the data distribution. Our key idea for generating novel samples of images is to interpolate between the clusters via a walk as per gradients of the dual function w.r.t. the data dimensions. In addition to the data efficiency gained from direct sampling, we propose an algorithm that offers a significant reduction in sample complexity for estimating the divergence of the data distribution with respect to the marginal distribution. We provide strong theoretical guarantees along with an extensive empirical evaluation using many real-world datasets from diverse domains, establishing the superiority of our approach w.r.t. state-of-the-art deep learning methods.

2024-04-10

ArXiv (preprint)

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Anthony

Timothee LESORT

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (see more)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (preprint)

Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok

Tikeng Notsawo Pascal Junior

Pascal Notsawo

Hattie Zhou

Mohammad Pezeshki

2024-03-04

ICLR.cc/2024/Workshop/ME-FoMo (poster)

Effective Latent Differential Equation Models via Attention and Multiple Shooting

Germán Abrevaya

Mahta Ramezanian-Panahi

Jean-Christophe Gagnon-Audet

Pablo Polosecki

Silvina Ponce Dawson

Guillermo Cecchi

2024-02-26

TMLR (accepted)

Unsupervised Concept Discovery Mitigates Spurious Correlations

Md Rifat Arefin

Yan Zhang

Aristide Baratin

Francesco Locatello

Dianbo Liu

Kenji Kawaguchi

2024-02-20

ArXiv (preprint)

Amplifying Pathological Detection in EEG Signaling Pathways through Cross-Dataset Transfer Learning

Mohammad-Javad Darvishi-Bayazi

Mohammad S. Ghaemi

Timothee LESORT

Md Rifat Arefin

Jocelyn Faubert

2024-02-01

Computers in Biology and Medicine (published)

Towards Machines that Trust: AI Agents Learn to Trust in the Trust Game

Ardavan S. Nobandegani

Thomas Shultz

Widely considered a cornerstone of human morality, trust shapes many aspects of human social interactions. In this work, we present a theore… (see more)tical analysis of the

2023-12-20

ArXiv (preprint)

Lag-Llama: Towards Foundation Models for Time Series Forecasting

Kashif Rasul

Arjun Ashok

Andrew Robert Williams

Arian Khorasani

George Adamopoulos

Rishika Bhagwatkar

Marin Biloš

Hena Ghonia

Nadhir Hassen

Anderson Schneider

Sahil Garg

Alexandre Drouin

Nicolas Chapados

Yuriy Nevmyvaka

Aiming to build foundation models for time-series forecasting and study their scaling behavior, we present here our work-in-progress on Lag-… (see more)Llama, a general-purpose univariate probabilistic time-series forecasting model trained on a large collection of time-series data. The model shows good zero-shot prediction capabilities on unseen "out-of-distribution" time-series datasets, outperforming supervised baselines. We use smoothly broken power-laws to fit and predict model scaling behavior. The open source code is made available at https://github.com/kashif/pytorch-transformer-ts.

2023-11-01

NeurIPS.cc/2023/Workshop/R0-FoMo (poster)

Gradient Masked Averaging for Federated Learning

Irene Tenison

Sai Aravind Sreeramadas

Vaikkunth Mugunthan

Edouard Oyallon

Federated learning (FL) is an emerging paradigm that permits a large number of clients with heterogeneous data to coordinate learning of a u… (see more)nified global model without the need to share data amongst each other. A major challenge in federated learning is the heterogeneity of data across client, which can degrade the performance of standard FL algorithms. Standard FL algorithms involve averaging of model parameters or gradient updates to approximate the global model at the server. However, we argue that in heterogeneous settings, averaging can result in information loss and lead to poor generalization due to the bias induced by dominant client gradients. We hypothesize that to generalize better across non-i.i.d datasets, the algorithms should focus on learning the invariant mechanism that is constant while ignoring spurious mechanisms that differ across clients. Inspired from recent works in Out-of-Distribution generalization, we propose a gradient masked averaging approach for FL as an alternative to the standard averaging of client updates. This aggregation technique for client updates can be adapted as a drop-in replacement in most existing federated algorithms. We perform extensive experiments on multiple FL algorithms with in-distribution, real-world, feature-skewed out-of-distribution, and quantity imbalanced datasets and show that it provides consistent improvements, particularly in the case of heterogeneous clients.

2023-10-23

TMLR (accepted)

Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting

Kashif Rasul

Arjun Ashok

Andrew Robert Williams

Arian Khorasani

George Adamopoulos

Rishika Bhagwatkar

Marin Bilovs

Hena Ghonia

Nadhir Hassen

Anderson Schneider

Sahil Garg

Alexandre Drouin

Nicolas Chapados

Yuriy Nevmyvaka

2023-10-12

ArXiv (preprint)

Comparison of Radiologists and Deep Learning for US Grading of Hepatic Steatosis.

Pedro Vianna

Sara-Ivana Calce

Pamela Boustros

Cassandra Larocque-Rigney

Laurent Patry-Beaudoin

Yi Hui Luo

Emre Aslan

John Marinos

Talal M. Alamri

Kim-Nhien Vu

Jessica Murphy-Lavallée

Jean-Sébastien Billiard

Emmanuel Montagnon

Hongliang Li

Samuel Kadoury

Bich Nguyen

Shanel Gauthier

Benjamin Thérien

Eugene Belilovsky … (see 4 more)

Guy Wolf

Michaël Chassé

Guy Cloutier

An Tang

Background Screening for nonalcoholic fatty liver disease (NAFLD) is suboptimal due to the subjective interpretation of US images. Purpose T… (see more)o evaluate the agreement and diagnostic performance of radiologists and a deep learning model in grading hepatic steatosis in NAFLD at US, with biopsy as the reference standard. Materials and Methods This retrospective study included patients with NAFLD and control patients without hepatic steatosis who underwent abdominal US and contemporaneous liver biopsy from September 2010 to October 2019. Six readers visually graded steatosis on US images twice, 2 weeks apart. Reader agreement was assessed with use of κ statistics. Three deep learning techniques applied to B-mode US images were used to classify dichotomized steatosis grades. Classification performance of human radiologists and the deep learning model for dichotomized steatosis grades (S0, S1, S2, and S3) was assessed with area under the receiver operating characteristic curve (AUC) on a separate test set. Results The study included 199 patients (mean age, 53 years ± 13 [SD]; 101 men). On the test set (n = 52), radiologists had fair interreader agreement (0.34 [95% CI: 0.31, 0.37]) for classifying steatosis grades S0 versus S1 or higher, while AUCs were between 0.49 and 0.84 for radiologists and 0.85 (95% CI: 0.83, 0.87) for the deep learning model. For S0 or S1 versus S2 or S3, radiologists had fair interreader agreement (0.30 [95% CI: 0.27, 0.33]), while AUCs were between 0.57 and 0.76 for radiologists and 0.73 (95% CI: 0.71, 0.75) for the deep learning model. For S2 or lower versus S3, radiologists had fair interreader agreement (0.37 [95% CI: 0.33, 0.40]), while AUCs were between 0.52 and 0.81 for radiologists and 0.67 (95% CI: 0.64, 0.69) for the deep learning model. Conclusion Deep learning approaches applied to B-mode US images provided comparable performance with human readers for detection and grading of hepatic steatosis. Published under a CC BY 4.0 license. Supplemental material is available for this article. See also the editorial by Tuthill in this issue.

2023-10-01

Radiology (published)

LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression

Ayush Kaushal

Tejas Vaidhya

Low Rank Decomposition of matrix - splitting a large matrix into a product of two smaller matrix offers a means for compression that reduces… (see more) the parameters of a model without sparsification, and hence delivering more speedup on modern hardware. Moreover, unlike quantization, the compressed linear layers remain fully differentiable and all the parameters trainable, while being able to leverage the existing highly efficient kernels over floating point matrices. We study the potential to compress Large Language Models (LLMs) for monolingual Code generation via Low Rank Decomposition (LoRD) and observe that ranks for the linear layers in these models can be reduced by upto 39.58% with less than 1% increase in perplexity. We then use Low Rank Decomposition (LoRD) to compress StarCoder 16B to 13.2B parameter with no drop and to 12.3B with minimal drop in HumanEval Pass@1 score, in less than 10 minutes on a single A100. The compressed models speeds up inference by up to 22.35% with just a single line of change in code over huggingface's implementation with pytorch backend. Low Rank Decomposition (LoRD) models remain compatible with state of the art near-lossless quantization method such as SpQR, which allows leveraging further compression gains of quantization. Lastly, QLoRA over Low Rank Decomposition (LoRD) model further reduces memory requirements by as much as 21.2% over vanilla QLoRA while offering similar gains from parameter efficient fine tuning. Our work shows Low Rank Decomposition (LoRD) as a promising new paradigm for LLM compression.

2023-09-25

ArXiv (preprint)