Sarath Chandar Anbil Parthipan

ali-rahimi.kalahroudi@mila.quebec

Biographie

Sarath Chandar est professeur adjoint à Polytechnique Montréal, où il dirige le laboratoire de recherche Chandar. Il est également membre du corps professoral de Mila – Institut québécois d’intelligence artificielle, et titulaire d'une chaire en IA Canada-CIFAR et d'une Chaire de recherche du Canada en apprentissage machine permanent. Ses recherches portent sur l'apprentissage tout au long de la vie, l'apprentissage profond, l'optimisation, l'apprentissage par renforcement et le traitement du langage naturel. Pour promouvoir la recherche sur l'apprentissage tout au long de la vie, Sarath Chandar a créé la Conférence sur les agents d'apprentissage tout au long de la vie (CoLLAs) en 2022 et a présidé le programme en 2022 et en 2023. Il est titulaire d'un doctorat de l'Université de Montréal et d'une maîtrise en recherche de l'Indian Institute of Technology Madras.

Étudiants actuels

Abdelrahman Zayed

Doctorat - Polytechnique Montréal

zayedabd@mila.quebec

Ali Rahimi-Kalahroudi

Maîtrise recherche - Université de Montréal

Amir Ardalan Kalantari Dehaghi

Collaborateur·rice alumni

Doctorat - Polytechnique Montréal

Co-superviseur⋅e :

Siva Reddy

andreas.madsen@mila.quebec

Maîtrise recherche - Polytechnique Montréal

antoine.clavaud@mila.quebec

Arjun Vaithilingam Sudhakar

Doctorat - Polytechnique Montréal

arjun.vaithilingam-sudhakar@mila.quebec

Artem Zholus

Doctorat - Polytechnique Montréal

artem.zholus@mila.quebec

Doctorat - Université de Montréal

darshan.patil@mila.quebec

francois.rivest@mila.quebec

Francois Rivest

Visiteur de recherche indépendant

goncalo-filipe.torcato-mordido@mila.quebec

Gabriele Prato

Doctorat - Université de Montréal

Postdoctorat - Polytechnique Montréal

Hadi NekoeiQachkanloo

Doctorat - Université de Montréal

nekoeihe@mila.quebec

Ista Abbes

Maîtrise recherche - Université de Montréal

istabrak.abbes@mila.quebec

Janarthanan Rajendran

Postdoctorat - Université de Montréal

Co-superviseur⋅e :

Doina Precup

janarthanan.rajendran@mila.quebec

jarrid.rector-brooks@mila.quebec

Jarrid Rector-Brooks

Doctorat - Université de Montréal

Superviseur⋅e principal⋅e :

Yoshua Bengio

Jerry Huang

Maîtrise recherche - Université de Montréal

jerry.huang@mila.quebec

Kshitij Gupta

Collaborateur·rice alumni - Université de Montréal

Superviseur⋅e principal⋅e :

Irina Rish

kshitij.gupta@mila.quebec

Lola Le Breton

Maîtrise recherche - Polytechnique Montréal

lola.lebreton@mila.quebec

maryam.hashemzadeh@mila.quebec

Louis Clouatre

Doctorat - Polytechnique Montréal

Superviseur⋅e principal⋅e :

Amal Zouaq

clouatrl@mila.quebec

Maryam Hashemzadeh

Doctorat - None

Mathieu Duchesneau

Doctorat - Université de Montréal

Postdoctorat

mathieu.reymond@mila.quebec

maziar.sargordi@mila.quebec

Maziar Sargordi

Doctorat - Polytechnique Montréal

Co-superviseur⋅e :

Amal Zouaq

mohammad-reza.samsami@mila.quebec

Megh Thakkar

Maîtrise recherche - Université de Montréal

megh.thakkar@mila.quebec

Mohammad R. Samsami

Maîtrise recherche - Université de Montréal

naga-karthik.enamundram@mila.quebec

Naga Karthik Enamundram

Doctorat - Polytechnique Montréal

Superviseur⋅e principal⋅e :

Julien Cohen-Adad

Doctorat - Polytechnique Montréal

pranshu.malviya@mila.quebec

prashant.govindarajan@mila.quebec

Prashant Govindarajan

Doctorat - Polytechnique Montréal

Simon Guiroy

Doctorat - Université de Montréal

Superviseur⋅e principal⋅e :

Xutong Zhao

Doctorat - Polytechnique Montréal

xutong.zhao@mila.quebec

Publications

Dealing With Non-stationarity in Decentralized Cooperative Multi-Agent Deep Reinforcement Learning via Multi-Timescale Learning

Hadi Nekoei

Akilesh Badrinaaraayanan

Amit Sinha

Mohammad Amin Amini

Janarthanan Rajendran

Aditya Mahajan

2023-11-20

Proceedings of The 2nd Conference on Lifelong Learning Agents (publié)

Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi

Hadi Nekoei

Xutong Zhao

Janarthanan Rajendran

Miao Liu

2023-11-20

Proceedings of The 2nd Conference on Lifelong Learning Agents (publié)

Language Model-In-The-Loop: Data Optimal Approach to Learn-To-Recommend Actions in Text Games

Arjun Vaithilingam Sudhakar

Prasanna Parthasarathi

Janarthanan Rajendran

2023-11-13

ArXiv (prépublication)

EpiK-Eval: Evaluation for Language Models as Epistemic Models

Gabriele Prato

Jerry Huang

Prasanna Parthasarathi

Shagun Sodhani

In the age of artificial intelligence, the role of large language models (LLMs) is becoming increasingly central. Despite their growing prev… (voir plus)alence, their capacity to consolidate knowledge from different training documents—a crucial ability in numerous applications—remains unexplored. This paper presents the first study examining the capability of LLMs to effectively combine such information within their parameter space. We introduce EpiK-Eval, a novel question-answering benchmark tailored to evaluate LLMs' proficiency in formulating a coherent and consistent knowledge representation from segmented narratives. Evaluations across various LLMs reveal significant weaknesses in this domain. We contend that these shortcomings stem from the intrinsic nature of prevailing training objectives. Consequently, we advocate for refining the approach towards knowledge consolidation, as it harbors the potential to dramatically improve their overall effectiveness and performance. The findings from this study offer insights for developing more robust and reliable LLMs. Our code and benchmark are available at https://github.com/chandar-lab/EpiK-Eval

2023-10-07

EMNLP/2023/Conference (accepté)

Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi

Hadi Nekoei

Xutong Zhao

Janarthanan Rajendran

Miao Liu

Cooperative Multi-agent Reinforcement Learning (MARL) algorithms with Zero-Shot Coordination (ZSC) have gained significant attention in rece… (voir plus)nt years. ZSC refers to the ability of agents to coordinate zero-shot (without additional interaction experience) with independently trained agents. While ZSC is crucial for cooperative MARL agents, it might not be possible for complex tasks and changing environments. Agents also need to adapt and improve their performance with minimal interaction with other agents. In this work, we show empirically that state-of-the-art ZSC algorithms have poor performance when paired with agents trained with different learning methods, and they require millions of interaction samples to adapt to these new partners. To investigate this issue, we formally defined a framework based on a popular cooperative multi-agent game called Hanabi to evaluate the adaptability of MARL methods. In particular, we created a diverse set of pre-trained agents and defined a new metric called adaptation regret that measures the agent's ability to efficiently adapt and improve its coordination performance when paired with some held-out pool of partners on top of its ZSC performance. After evaluating several SOTA algorithms using our framework, our experiments reveal that naive Independent Q-Learning (IQL) agents in most cases adapt as quickly as the SOTA ZSC algorithm Off-Belief Learning (OBL). This finding raises an interesting research question: How to design MARL algorithms with high ZSC performance and capability of fast adaptation to unseen partners. As a first step, we studied the role of different hyper-parameters and design choices on the adaptability of current MARL algorithms. Our experiments show that two categories of hyper-parameters controlling the training data diversity and optimization process have a significant impact on the adaptability of Hanabi agents.

2023-08-20

ArXiv (prépublication)

Thompson Sampling for Improved Exploration in GFlowNets

Jarrid Rector-Brooks

Kanika Madan

Moksh J. Jain

Maksym Korablyov

Cheng-Hao Liu

Nikolay Malkin

Yoshua Bengio

Generative flow networks (GFlowNets) are amortized variational inference algorithms that treat sampling from a distribution over composition… (voir plus)al objects as a sequential decision-making problem with a learnable action policy. Unlike other algorithms for hierarchical sampling that optimize a variational bound, GFlowNet algorithms can stably run off-policy, which can be advantageous for discovering modes of the target distribution. Despite this flexibility in the choice of behaviour policy, the optimal way of efficiently selecting trajectories for training has not yet been systematically explored. In this paper, we view the choice of trajectories for training as an active learning problem and approach it using Bayesian techniques inspired by methods for multi-armed bandits. The proposed algorithm, Thompson sampling GFlowNets (TS-GFN), maintains an approximate posterior distribution over policies and samples trajectories from this posterior for training. We show in two domains that TS-GFN yields improved exploration and thus faster convergence to the target distribution than the off-policy exploration strategies used in past work.

2023-06-19

ICML.cc/2023/Workshop/SPIGM (poster)

Should We Attend More or Less? Modulating Attention for Fairness

A. Zayed

Goncalo Mordido

Samira Shabanian

2023-05-22

ArXiv (prépublication)

Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learning

Xutong Zhao

Yangchen Pan

Chenjun Xiao

Janarthanan Rajendran

Efficient exploration is critical in cooperative deep Multi-Agent Reinforcement Learning (MARL). In this work, we propose an exploration met… (voir plus)hod that effectively encourages cooperative exploration based on the idea of sequential action-computation scheme. The high-level intuition is that to perform optimism-based exploration, agents would explore cooperative strategies if each agent's optimism estimate captures a structured dependency relationship with other agents. Assuming agents compute actions following a sequential order at \textit{each environment timestep}, we provide a perspective to view MARL as tree search iterations by considering agents as nodes at different depths of the search tree. Inspired by the theoretically justified tree search algorithm UCT (Upper Confidence bounds applied to Trees), we develop a method called Conditionally Optimistic Exploration (COE). COE augments each agent's state-action value estimate with an action-conditioned optimistic bonus derived from the visitation count of the global state and joint actions of preceding agents. COE is performed during training and disabled at deployment, making it compatible with any value decomposition method for centralized training with decentralized execution. Experiments across various cooperative MARL benchmarks show that COE outperforms current state-of-the-art exploration methods on hard-exploration tasks.

2023-05-08

auai.org/UAI/2023/Conference (publié)

Conditionally optimistic exploration for cooperative deep multi-agent reinforcement learning

Xutong Zhao

Yangchen Pan

Chenjun Xiao

Janarthanan Rajendran

Efficient exploration is critical in cooperative deep Multi-Agent Reinforcement Learning (MARL). In this work, we propose an exploration met… (voir plus)hod that effectively encourages cooperative exploration based on the idea of sequential action-computation scheme. The high-level intuition is that to perform optimism-based exploration, agents would explore cooperative strategies if each agent’s optimism estimate captures a structured dependency relationship with other agents. Assuming agents compute actions following a sequential order at each environment timestep, we provide a perspective to view MARL as tree search iterations by considering agents as nodes at different depths of the search tree. Inspired by the theoretically justified tree search algorithm UCT (Upper Confidence bounds applied to Trees), we develop a method called Conditionally Optimistic Exploration (COE). COE augments each agent’s state-action value estimate with an action-conditioned optimistic bonus derived from the visitation count of the global state and joint actions of preceding agents. COE is performed during training and disabled at deployment, making it compatible with any value decomposition method for centralized training with decentralized execution. Experiments across various cooperative MARL benchmarks show that COE outperforms current state-of-the-art exploration methods on hard-exploration tasks.

2023-05-08

auai.org/UAI/2023/Conference (publié)

Behavioral Cloning for Crystal Design

Prashant Govindarajan

Santiago Miret

Jarrid Rector-Brooks

Mariano Phielipp

Janarthanan Rajendran

Solid-state materials, which are made up of periodic 3D crystal structures, are particularly useful for a variety of real-world applications… (voir plus) such as batteries, fuel cells and catalytic materials. Designing solid-state materials, especially in a robust and automated fashion, remains an ongoing challenge. To further the automated design of crystalline materials, we propose a method to learn to design valid crystal structures given a crystal skeleton. By incorporating Euclidean equivariance into a policy network, we portray the problem of designing new crystals as a sequential prediction task suited for imitation learning. At each step, given an incomplete graph of a crystal skeleton, an agent assigns an element to a specific node. We adopt a behavioral cloning strategy to train the policy network on data consisting of curated trajectories generated from known crystals.

2023-03-17

ICLR.cc/2023/Workshop/ML4Materials (poster)

Dealing With Non-stationarity in Decentralized Cooperative Multi-Agent Deep Reinforcement Learning via Multi-Timescale Learning

Hadi Nekoei

Akilesh Badrinaaraayanan

Amit Sinha

Mohammad Amin Amini

Janarthanan Rajendran

Aditya Mahajan

2023-02-06

ArXiv (prépublication)

An Empirical Investigation of the Role of Pre-training in Lifelong Learning

Sanket Vaibhav Mehta

Darshan Patil

Emma Strubell

The lifelong learning paradigm in machine learning is an attractive alternative to the more prominent isolated learning scheme not only due … (voir plus)to its resemblance to biological learning, but also its potential to reduce energy waste by obviating excessive model re-training. A key challenge to this paradigm is the phenomenon of catastrophic forgetting. With the increasing popularity and success of pre-trained models in machine learning, we pose the question: What role does pre-training play in lifelong learning, specifically with respect to catastrophic forgetting? We investigate existing methods in the context of large, pre-trained models and evaluate their performance on a variety of text and image classification tasks, including a large-scale study using a novel dataset of 15 diverse NLP tasks. Across all settings, we observe that generic pre-training implicitly alleviates the effects of catastrophic forgetting when learning multiple tasks sequentially compared to randomly initialized models. We then further investigate why pre-training alleviates forgetting in this setting. We study this phenomenon by analyzing the loss landscape, finding that pre-trained weights appear to ease forgetting by leading to wider minima. Based on this insight, we propose jointly optimizing for current task loss and loss basin sharpness in order to explicitly encourage wider basins during sequential fine-tuning. We show that this optimization approach leads to performance comparable to the state-of-the-art in task-sequential continual learning across multiple settings, without retaining a memory that scales in size with the number of tasks.