Portrait of Irina Rish

Irina Rish

Core Academic Member

irina.rish@mila.quebec

Canada CIFAR AI Chair

Full Professor, Université de Montréal, Department of Computer Science and Operations Research Department

Research Topics

Computational Neuroscience

Deep Learning

Generative Models

Multimodal Learning

Natural Language Processing

Online Learning

Reinforcement Learning

Biography

Irina Rish is a full professor at the Université de Montréal (UdeM), where she leads the Autonomous AI Lab, and a core academic member of Mila – Quebec Artificial Intelligence Institute.

In addition to holding a Canada Excellence Research Chair (CERC) and a CIFAR Chair, she leads the U.S. Department of Energy’s INCITE project on Scalable Foundation Models on Summit & Frontier supercomputers at the Oak Ridge Leadership Computing Facility. She co-founded and serves as CSO of Nolano.ai.

Rish’s current research interests include neural scaling laws and emergent behaviors (capabilities and alignment) in foundation models, as well as continual learning, out-of-distribution generalization and robustness.

Before joining UdeM in 2019, she was a research scientist at the IBM T.J. Watson Research Center, where she worked on various projects at the intersection of neuroscience and AI, and led the Neuro-AI challenge. She was awarded the IBM Eminence & Excellence Award and IBM Outstanding Innovation Award (2018), IBM Outstanding Technical Achievement Award (2017) and IBM Research Accomplishment Award (2009).

She holds 64 patents and has published 120 research papers, several book chapters, three edited books and a monograph on sparse modeling.

Current Students

George Adamopoulos

Research Intern

PhD - Université de Montréal

Co-supervisor :

Samira Ebrahimi Kahou

PhD - Université de Montréal

PhD - Université de Montréal

Co-supervisor :

Alexandre Drouin

Rishika Bhagwatkar

Master's Research - Université de Montréal

Ethan Caballero

PhD - McGill University

Principal supervisor :

PhD - Université de Montréal

Mohammad Javad Darvishi Bayazi

PhD - Université de Montréal

PhD - Université de Montréal

Co-supervisor :

Master's Research - Concordia University

Principal supervisor :

Mirco Ravanelli

Mojtaba Faramarzi

PhD - Université de Montréal

Collaborating Alumni - Université de Montréal

Master's Research

PhD - Université de Montréal

Principal supervisor :

Ioannis Mitliagkas

PhD - Université de Montréal

Principal supervisor :

Eilif Benjamin Muller

Arnav Kumar Jain

PhD - Université de Montréal

Collaborating researcher

Master's Research - Concordia University

Principal supervisor :

Eugene Belilovsky

Arian Khorasani

Master's Research - Université de Montréal

Professional Master's - Université de Montréal

PhD - Concordia University

Principal supervisor :

Eugene Belilovsky

Master's Research - Université de Montréal

Collaborating researcher

Ekaterina Lobacheva

Postdoctorate - Université de Montréal

Principal supervisor :

Nicolas Le Roux

Master's Research - Université de Montréal

amin.mansouri@mila.quebec

Collaborating Alumni

PhD - Université de Montréal

Master's Research - Université de Montréal

Master's Research - Université de Montréal

PhD - Université de Montréal

Co-supervisor :

Eilif Benjamin Muller

Mohammad Pezeshki

Collaborating researcher

Co-supervisor :

Guillaume Lajoie

Motahareh Pourrahimi

PhD - McGill University

Principal supervisor :

Mahta Ramezanian

Master's Research - Université de Montréal

Co-supervisor :

Guillaume Dumas

Collaborating researcher - Université de Montréal

PhD - Université de Montréal

PhD - McGill University

Principal supervisor :

Zahra Sheikhbahaee

Postdoctorate - Université de Montréal

Principal supervisor :

Guillaume Dumas

PhD - Concordia University

Principal supervisor :

Eugene Belilovsky

Gopeshh Subbaraj

PhD - Université de Montréal

Benjamin Therien

PhD - Université de Montréal

Co-supervisor :

Eugene Belilovsky

Master's Research - Université de Montréal

Andrew Williams

PhD - Université de Montréal

Co-supervisor :

Nicolas Chapados

Master's Research - Université de Montréal

Publications

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Anthony

Timothee LESORT

Eugene Belilovsky

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (see more)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (preprint)

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Gregory Anthony

Timothee LESORT

Eugene Belilovsky

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (see more)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (preprint)

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Gregory Anthony

Timothee LESORT

Eugene Belilovsky

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (see more)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (preprint)

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Gregory Anthony

Timothee LESORT

Eugene Belilovsky

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (see more)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (preprint)

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Gregory Anthony

Timothee LESORT

Eugene Belilovsky

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (see more)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (preprint)

Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok

Tikeng Notsawo Pascal Junior

Pascal Notsawo

Hattie Zhou

Mohammad Pezeshki

Guillaume Dumas

2024-03-04

ICLR.cc/2024/Workshop/ME-FoMo (poster)

Effective Latent Differential Equation Models via Attention and Multiple Shooting

Germán Abrevaya

Mahta Ramezanian-Panahi

Jean-Christophe Gagnon-Audet

Pablo Polosecki

Silvina Ponce Dawson

Guillermo Cecchi

Guillaume Dumas

2024-02-26

TMLR (accepted)

Amplifying Pathological Detection in EEG Signaling Pathways through Cross-Dataset Transfer Learning

Mohammad Javad Darvishi Bayazi

Mohammad S. Ghaemi

Timothee LESORT

Md Rifat Arefin

Jocelyn Faubert

2024-02-01

Computers in Biology and Medicine (published)

Dance of the Neurons: Unraveling Sex from Brain Signals (short paper).

Mohammad-Javad Darvishi Bayazi

Mohammad S. Ghaemi

Jocelyn Faubert

2024-01-01

ML4CMH@AAAI (published)

dblp.uni-trier.de

Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design.

Rishika Bhagwatkar

Shravan Nayak

2024-01-01

EMNLP (Findings) (published)

Towards Machines that Trust: AI Agents Learn to Trust in the Trust Game

Ardavan S. Nobandegani

Thomas Shultz

Widely considered a cornerstone of human morality, trust shapes many aspects of human social interactions. In this work, we present a theore… (see more)tical analysis of the

2023-12-20

ArXiv (preprint)

Challenging Common Assumptions about Catastrophic Forgetting and Knowledge Accumulation

Timothee LESORT

Oleksiy Ostapenko

Pau Rodriguez

Diganta Misra

Md Rifat Arefin

Laurent Charlin

2023-11-20

Proceedings of The 2nd Conference on Lifelong Learning Agents (published)

proceedings.mlr.press