Portrait de Irina Rish

Irina Rish

Membre académique principal

irina.rish@mila.quebec

Chaire en IA Canada-CIFAR

Professeure titulaire, Université de Montréal, Département d'informatique et de recherche opérationnelle

Sujets de recherche

Apprentissage en ligne

Apprentissage multimodal

Apprentissage par renforcement

Apprentissage profond

Modèles génératifs

Neurosciences computationnelles

Traitement du langage naturel

Biographie

Irina Rish est professeure titulaire à l'Université de Montréal (UdeM), où elle dirige le Laboratoire d'IA autonome. Membre du corps professoral de Mila – Institut québécois d’intelligence artificielle, elle est titulaire d'une chaire d'excellence en recherche du Canada (CERC) et d'une chaire en IA Canada-CIFAR. Irina dirige le projet INCITE du ministère américain de l'Environnement au sujet des modèles de fondation évolutifs sur les superordinateurs Summit et Frontier à l'Oak Ridge Leadership Computing Facility (OLCF). Elle est cofondatrice et directrice scientifique de Nolano.ai.

Ses recherches actuelles portent sur les lois de mise à l'échelle neuronale et les comportements émergents (capacités et alignement) dans les modèles de fondation, ainsi que sur l'apprentissage continu, la généralisation hors distribution et la robustesse. Avant de se joindre à l'UdeM en 2019, Irina était chercheuse au Centre de recherche IBM Thomas J. Watson, où elle a travaillé sur divers projets à l'intersection des neurosciences et de l'IA, et dirigé le défi NeuroAI. Elle a reçu plusieurs prix IBM : ceux de l’excellence et de l’innovation exceptionnelle (2018), celui de la réalisation technique exceptionnelle (2017), et celui de l’accomplissement en recherche (2009). Elle détient 64 brevets et a écrit plus de 120 articles de recherche, plusieurs chapitres de livres, trois livres publiés et une monographie sur la modélisation éparse.

Étudiants actuels

George Adamopoulos

Stagiaire de recherche

Doctorat - UdeM

Co-superviseur⋅e :

Samira Ebrahimi Kahou

Doctorat - UdeM

Doctorat - UdeM

Co-superviseur⋅e :

Alexandre Drouin

Rishika Bhagwatkar

Maîtrise recherche - UdeM

Ethan Caballero

Doctorat - McGill

Superviseur⋅e principal⋅e :

Doctorat - UdeM

Mohammad Javad Darvishi Bayazi

Doctorat - UdeM

Doctorat - UdeM

Co-superviseur⋅e :

Maîtrise recherche - Concordia

Superviseur⋅e principal⋅e :

Mirco Ravanelli

Mojtaba Faramarzi

Doctorat - UdeM

Collaborateur·rice alumni - UdeM

Maîtrise recherche

Doctorat - UdeM

Superviseur⋅e principal⋅e :

Ioannis Mitliagkas

Doctorat - UdeM

Superviseur⋅e principal⋅e :

Eilif Benjamin Muller

Arnav Kumar Jain

Doctorat - UdeM

Collaborateur·rice de recherche

Maîtrise recherche - Concordia

Superviseur⋅e principal⋅e :

Eugene Belilovsky

Arian Khorasani

Maîtrise recherche - UdeM

Maîtrise professionnelle - UdeM

Doctorat - Concordia

Superviseur⋅e principal⋅e :

Eugene Belilovsky

Maîtrise recherche - UdeM

Collaborateur·rice de recherche

Maîtrise recherche - UdeM

amin.mansouri@mila.quebec

Collaborateur·rice alumni

Doctorat - UdeM

Maîtrise recherche - UdeM

Maîtrise recherche - UdeM

Doctorat - UdeM

Co-superviseur⋅e :

Eilif Benjamin Muller

Mohammad Pezeshki

Collaborateur·rice de recherche

Co-superviseur⋅e :

Guillaume Lajoie

Motahareh Pourrahimi

Doctorat - McGill

Superviseur⋅e principal⋅e :

Mahta Ramezanian

Maîtrise recherche - UdeM

Co-superviseur⋅e :

Guillaume Dumas

Collaborateur·rice de recherche - UdeM

Doctorat - UdeM

Doctorat - McGill

Superviseur⋅e principal⋅e :

Zahra Sheikhbahaee

Postdoctorat - UdeM

Superviseur⋅e principal⋅e :

Guillaume Dumas

Doctorat - Concordia

Superviseur⋅e principal⋅e :

Eugene Belilovsky

Gopeshh Subbaraj

Doctorat - UdeM

Benjamin Therien

Doctorat - UdeM

Co-superviseur⋅e :

Eugene Belilovsky

Maîtrise recherche - UdeM

Andrew Williams

Doctorat - UdeM

Co-superviseur⋅e :

Nicolas Chapados

Maîtrise recherche - UdeM

Publications

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Gregory Anthony

Timothee LESORT

Eugene Belilovsky

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (voir plus)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (prépublication)

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Gregory Anthony

Timothee LESORT

Eugene Belilovsky

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (voir plus)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (prépublication)

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Anthony

Timothee LESORT

Eugene Belilovsky

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (voir plus)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (prépublication)

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Gregory Anthony

Timothee LESORT

Eugene Belilovsky

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (voir plus)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (prépublication)

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Adam Ibrahim

Benjamin Thérien

Kshitij Gupta

Mats Leon Richter

Quentin Gregory Anthony

Timothee LESORT

Eugene Belilovsky

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes ava… (voir plus)ilable. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English

2024-03-13

ArXiv (prépublication)

Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok

Tikeng Notsawo Pascal Junior

Pascal Notsawo

Hattie Zhou

Mohammad Pezeshki

Guillaume Dumas

2024-03-04

ICLR.cc/2024/Workshop/ME-FoMo (poster)

Effective Latent Differential Equation Models via Attention and Multiple Shooting

Germán Abrevaya

Mahta Ramezanian-Panahi

Jean-Christophe Gagnon-Audet

Pablo Polosecki

Silvina Ponce Dawson

Guillermo Cecchi

Guillaume Dumas

2024-02-26

TMLR (accepté)

Amplifying Pathological Detection in EEG Signaling Pathways through Cross-Dataset Transfer Learning

Mohammad Javad Darvishi Bayazi

Mohammad S. Ghaemi

Timothee LESORT

Md Rifat Arefin

Jocelyn Faubert

2024-02-01

Computers in Biology and Medicine (publié)

Dance of the Neurons: Unraveling Sex from Brain Signals (short paper).

Mohammad-Javad Darvishi Bayazi

Mohammad S. Ghaemi

Jocelyn Faubert

2024-01-01

ML4CMH@AAAI (publié)

dblp.uni-trier.de

Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design.

Rishika Bhagwatkar

Shravan Nayak

2024-01-01

EMNLP (Findings) (publié)

Towards Machines that Trust: AI Agents Learn to Trust in the Trust Game

Ardavan S. Nobandegani

Thomas Shultz

Widely considered a cornerstone of human morality, trust shapes many aspects of human social interactions. In this work, we present a theore… (voir plus)tical analysis of the

2023-12-20

ArXiv (prépublication)

Challenging Common Assumptions about Catastrophic Forgetting and Knowledge Accumulation

Timothee LESORT

Oleksiy Ostapenko

Pau Rodriguez

Diganta Misra

Md Rifat Arefin

Laurent Charlin

2023-11-20

Proceedings of The 2nd Conference on Lifelong Learning Agents (publié)

proceedings.mlr.press