Sarath Chandar

Biography

Sarath Chandar is an associate professor at Polytechnique Montreal's Department of Computer and Software Engineering, where he leads the Chandar Research Lab. He is also a Core Academic Member at Mila – Quebec Artificial Intelligence Institute and holds a Canada CIFAR AI Chair and the Canada Research Chair in Lifelong Machine Learning.

Chandar’s research interests include lifelong learning, deep learning, optimization, reinforcement learning and natural language processing. To promote research in lifelong learning, Chandar created the Conference on Lifelong Learning Agents (CoLLAs) in 2022, for which he served as program chair in 2022 and 2023.

He has a PhD from Université de Montréal and an MSc (By Research) from the Indian Institute of Technology Madras.

Current Students

Ista Abbes

Master's Research - Université de Montréal

Davide Baldelli

PhD - Polytechnique Montréal

Co-supervisor :

Master's Research - Polytechnique Montréal

Louis Clouatre

PhD - Polytechnique Montréal

Principal supervisor :

Amal Zouaq

Naga Karthik Enamundram

PhD - Polytechnique Montréal

Principal supervisor :

Julien Cohen-Adad

emvnagakarthik@gmail.com

Prashant Govindarajan

PhD - Polytechnique Montréal

Simon Guiroy

PhD - Université de Montréal

Principal supervisor :

Collaborating researcher - Université de Montréal

Principal supervisor :

Liam Paull

Maryam Hashemzadeh

PhD - Université de Montréal

David Heurtel--Depeiges

PhD - Polytechnique Montréal

Amir Ardalan Kalantari Dehaghi

Jerry Huang

PhD - Université de Montréal

Collaborating Alumni

Lola Le Breton

Master's Research - Polytechnique Montréal

andreas.madsen@mila.quebec

Andreas Madsen

PhD - Polytechnique Montréal

Co-supervisor :

Siva Reddy

PhD - Polytechnique Montréal

Master's Research - Polytechnique Montréal

Mohamed Amine Merzouk

Postdoctorate - Polytechnique Montréal

Principal supervisor :

Hadi NekoeiQachkanloo

PhD - Université de Montréal

Darshan Patil

PhD - Université de Montréal

Gabriele Prato

PhD - Université de Montréal

Janarthanan Rajendran

Collaborating Alumni - Université de Montréal

Co-supervisor :

Postdoctorate

Independent visiting researcher

Arjun Vaithilingam Sudhakar

Mohammad R. Samsami

Master's Research - Université de Montréal

Master's Research - Université de Montréal

PhD - Polytechnique Montréal

Abdelrahman Zayed

PhD - Polytechnique Montréal

Xutong Zhao

PhD - Polytechnique Montréal

Artem Zholus

PhD - Polytechnique Montréal

NeoBERT: A New Frontier for Open-Source Encoder Language Models

Blog Posts

A digital picture of Bert from Sesame street, wering black trench coat and sunglasses

March 3, 2025

Lola Le Breton

Quentin Fournier

Sarath Chandar

Read the article

October 1, 2024

How Do We Explain AI and Ensure the Explanation Is True? Faithfulness Measurable Models Tell You How

Andrea Madsen

Siva Reddy

Sarath Chandar

Read the article

Publications

Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Models

Jerry Huang

Prasanna Parthasarathi

Mehdi Rezagholizadeh

2024-01-01

EMNLP (published)

Exploring Quantization for Efficient Pre-Training of Transformer Language Models

Kamran Chitsaz

Quentin Fournier

Goncalo Mordido

The increasing scale of Transformer models has led to an increase in their pre-training computational requirements. While quantization has p… (see more)roven to be effective after pre-training and during fine-tuning, applying quantization in Transformers during pre-training has remained largely unexplored at scale for language modeling. This study aims to explore the impact of quantization for efficient pre-training of Transformers, with a focus on linear layer components. By systematically applying straightforward linear quantization to weights, activations, gradients, and optimizer states, we assess its effects on model efficiency, stability, and performance during training. By offering a comprehensive recipe of effective quantization strategies to be applied during the pre-training of Transformers, we promote high training efficiency from scratch while retaining language modeling ability. Code is available at https://github.com/chandar-lab/EfficientLLMs.

2024-01-01

EMNLP (Findings) (published)

Do Large Language Models Know How Much They Know?

Gabriele Prato

Jerry Huang

Prasanna Parthasarathi

Shagun Sodhani

2024-01-01

EMNLP (published)

Do Large Language Models Know How Much They Know?

Gabriele Prato

Jerry Huang

Prasanna Parthasarathi

Shagun Sodhani

Large Language Models (LLMs) have emerged as highly capable systems and are increasingly being integrated into various uses. Nevertheless, t… (see more)he rapid advancement in their deployment trails a comprehensive understanding of their internal mechanisms, as well as a delineation of their capabilities and limitations. A desired characteristic of an intelligent system is its ability to recognize the scope of its own knowledge. To investigate whether LLMs embody this attribute, we develop a benchmark that challenges these models to enumerate all information they possess on specific topics. This benchmark assesses whether the models recall excessive, insufficient, or the precise amount of required information, thereby indicating their awareness of how much they know about the given topic. Our findings reveal that the emergence of this property varies across different architectures and manifests at diverse rates. However, with sufficient scaling, all tested models are ultimately capable of performing this task. The insights gained from this research advance our understanding of LLMs, shedding light on their operational capabilities and contributing to the ongoing exploration of their intricate dynamics.

2024-01-01

Conference on Empirical Methods in Natural Language Processing (published)

www.semanticscholar.org

Learning Conditional Policies for Crystal Design Using Offline Reinforcement Learning

Prashant Govindarajan

Santiago Miret

Jarrid Rector-Brooks

Mariano Phielipp

Janarthanan Rajendran

Navigating through the exponentially large chemical space to search for desirable materials is an extremely challenging task in material dis… (see more)covery. Recent developments in generative and geometric deep learning have shown...

2024-01-01

Digital Discovery (published)

openreview.net

Learning Conditional Policies for Crystal Design Using Offline Reinforcement Learning

Prashant Govindarajan

Santiago Miret

Jarrid Rector-Brooks

Mariano Phielipp

Janarthanan Rajendran

2024-01-01

Digital Discovery (published)

MVP: Minimal Viable Phrase for Long Text Understanding.

Louis Clouâtre

Amal Zouaq

2024-01-01

International Conference on Language Resources and Evaluation (published)

dblp.uni-trier.de

Fairness-Aware Structured Pruning in Transformers

Abdelrahman Zayed

Goncalo Mordido

Samira Shabanian

Ioana Baldini

The increasing size of large language models (LLMs) has introduced challenges in their training and inference. Removing model components is … (see more)perceived as a solution to tackle the large model sizes, however, existing pruning methods solely focus on performance, without considering an essential aspect for the responsible use of LLMs: model fairness. It is crucial to address the fairness of LLMs towards diverse groups, such as women, Black people, LGBTQ+, Jewish communities, among others, as they are being deployed and available to a wide audience. In this work, first, we investigate how attention heads impact fairness and performance in pre-trained transformer-based language models. We then propose a novel method to prune the attention heads that negatively impact fairness while retaining the heads critical for performance, i.e. language modeling capabilities. Our approach is practical in terms of time and resources, as it does not require fine-tuning the final pruned, and fairer, model. Our findings demonstrate a reduction in gender bias by 19%, 19.5%, 39.5%, 34.7%, 23%, and 8% for DistilGPT-2, GPT-2, GPT-Neo of two different sizes, GPT-J, and Llama 2 models, respectively, in comparison to the biased model, with only a slight decrease in performance. WARNING: This work uses language that is offensive in nature.

2023-12-24

ArXiv (preprint)

Fairness-Aware Structured Pruning in Transformers

Abdelrahman Zayed

Goncalo Mordido

Samira Shabanian

Ioana Baldini