Home

Inspiring the development of artificial intelligence for the benefit of all 

A professor talks to his students in a café/lounge.

Located in the heart of Quebec’s AI ecosystem, Mila is a community of more than 1,200 researchers specializing in machine learning and dedicated to scientific excellence and innovation.

About

Featured

News

09 Jul 2024
AI4GoodLab 2024: 17 Projects Harnessing AI for Social Good
2024 cohort of the AI4Good Lab program.
Read the story
03 Jul 2024
Mila and Sorintellis Announce a Partnership to Optimize Complex Decision-Making Processes in Clinical Drug Development
The Mila logo and the Sorintellis logo.
Read the story
19 Jun 2024
Introducing the First Cohort of the Indigenous Pathfinders in AI Program
Visual identity of the Indigenous Pathfinders in AI program.
Read the story
See more news

Faculty 

Founded in 1993 by Professor Yoshua Bengio, Mila today brings together over 130 professors affiliated with Université de Montréal, McGill University, Polytechnique Montréal and HEC Montréal. Mila also welcomes professors from Université Laval, Université de Sherbrooke and École de technologie supérieure (ÉTS). 

Browse the online directory

Photo of Yoshua Bengio

Latest Publications

Group Membership Bias
Ali Vardasbi
Maarten de Rijke
Fernando Diaz
Mostafa Dehghani
2024-07-11
Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (published)
doi.org
arxiv.org
Simple and Scalable Strategies to Continually Pre-train Large Language Models
Adam Ibrahim
Benjamin Thérien
Kshitij Gupta
Mats Leon Richter
Quentin Gregory Anthony
Timothee LESORT
Eugene Belilovsky
Irina Rish
2024-07-08
TMLR (accepted)
doi.org
openreview.net
LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression
Ayush Kaushal
Tejas Vaidhya
Irina Rish
Low Rank Decomposition of matrix - splitting a large matrix into a product of two smaller matrix offers a means for compression that reduces… (see more) the parameters of a model without sparsification, and hence delivering more speedup on modern hardware. Moreover, unlike quantization, the compressed linear layers remain fully differentiable and all the parameters trainable, while being able to leverage the existing highly efficient kernels over floating point matrices. We study the potential to compress Large Language Models (LLMs) for monolingual Code generation via Low Rank Decomposition (LoRD) and observe that ranks for the linear layers in these models can be reduced by upto 39.58% with less than 1% increase in perplexity. We then use Low Rank Decomposition (LoRD) to compress StarCoder 16B to 13.2B parameter with no drop and to 12.3B with minimal drop in HumanEval Pass@1 score, in less than 10 minutes on a single A100. The compressed models speeds up inference by up to 22.35% with just a single line of change in code over huggingface's implementation with pytorch backend. Low Rank Decomposition (LoRD) models remain compatible with state of the art near-lossless quantization method such as SpQR, which allows leveraging further compression gains of quantization. Lastly, QLoRA over Low Rank Decomposition (LoRD) model further reduces memory requirements by as much as 21.2% over vanilla QLoRA while offering similar gains from parameter efficient fine tuning. Our work shows Low Rank Decomposition (LoRD) as a promising new paradigm for LLM compression.
2024-07-03
ICML.cc/2024/Workshop/FM-Wild (poster)
doi.org
openreview.net
A Bayesian Non-Stationary Heteroskedastic Time Series Model for Multivariate Critical Care Data
Zayd Omar
David A. Stephens
Alexandra M. Schmidt
David Buckeridge
2024-07-02
Statistics in Medicine (published)
doi.org
arxiv.org
See more publications

AI for Humanity

Socially responsible and beneficial development of AI is a fundamental component of Mila’s mission. As a leader in the field, we wish to contribute to social dialogue and the development of applications that will benefit society.

Learn more

A person looks up at a starry sky.