Irina Rish

Ivan Anokhin

Doctorat - UdeM

Co-superviseur⋅e :

Samira Ebrahimi Kahou

ivan.anokhin@mila.quebec

Doctorat - UdeM

Arjun Ashok

Doctorat

Co-superviseur⋅e :

Alexandre Drouin

arjun.ashok@mila.quebec

Reza Bayat

Maîtrise recherche - UdeM

Co-superviseur⋅e :

Pouya Bashivan

reza.bayat@mila.quebec

rishika.bhagwatkar@mila.quebec

Rishika Bhagwatkar

Maîtrise recherche - UdeM

Collaborateur·rice de recherche - UdeM

connor.brennan@mila.quebec

Ethan Caballero

Doctorat - McGill

Superviseur⋅e principal⋅e :

Blake Richards

ethan.caballero@mila.quebec

paolo.cudrano@mila.quebec

Paolo Cudrano

Collaborateur·rice de recherche - Politecnico di Milano

Mohammad-Javad Darvishi Bayazi

Amin Darabi

Doctorat - UdeM

amin.darabi@mila.quebec

Doctorat - UdeM

mohammad-javad.darvishi-bayasi@mila.quebec

Doctorat - UdeM

Co-superviseur⋅e :

Doctorat - UdeM

Collaborateur·rice alumni - UdeM

Co-superviseur⋅e :

Sarath Chandar Anbil Parthipan

kshitij.gupta@mila.quebec

Nadhir Hassen

Stagiaire de recherche - UdeM

nadhir.hassen@mila.quebec

Prateek Humane

Collaborateur·rice de recherche

prateek.humane@mila.quebec

Doctorat - UdeM

Superviseur⋅e principal⋅e :

Nizar Islah

Doctorat - UdeM

Superviseur⋅e principal⋅e :

Eilif Benjamin Muller

nizar.islah@mila.quebec

Doctorat - UdeM

arnav-kumar.jain@mila.quebec

Collaborateur·rice de recherche

ayush.kaushal@mila.quebec

Arian Khorasani

Maîtrise recherche - UdeM

arian.khorasani@mila.quebec

Neeraj Kumar

Maîtrise professionnelle - UdeM

neeraj.kumar@mila.quebec

Gwen Legate

Doctorat - Concordia

Superviseur⋅e principal⋅e :

gwendolyne.legate@mila.quebec

ekaterina.lobacheva@mila.quebec

Kate Lobacheva

Postdoctorat

Superviseur⋅e principal⋅e :

Nicolas Le Roux

Maîtrise recherche - UdeM

sayed.mansouri-tehrani@mila.quebec

juan.mayor-torres@mila.quebec

Juan Manuel Mayor-Torres

Collaborateur·rice de recherche

Visiteur de recherche indépendant

memariaa@mila.quebec

Andrei Mircea Romascanu

Doctorat - UdeM

Maîtrise recherche - UdeM

sparsha.mishra@mila.quebec

Diganta Misra

Maîtrise recherche - UdeM

diganta.misra@mila.quebec

motahareh.pourrahimi@mila.quebec

Remus Mocanu

Maîtrise recherche - UdeM

remus.mocanu@mila.quebec

Timothy Nest

Doctorat - UdeM

Co-superviseur⋅e :

Eilif Benjamin Muller

timothy.nest@mila.quebec

Mohammad Pezeshki

Collaborateur·rice de recherche

Doctorat - McGill

Superviseur⋅e principal⋅e :

Pouya Bashivan

Maximilian Puelma Touzel

Collaborateur·rice de recherche

Maîtrise recherche - UdeM

Co-superviseur⋅e :

mahta.ramezanian@mila.quebec

Roland Riachi

Collaborateur·rice de recherche - UdeM

roland.riachi@mila.quebec

Matthew Riemer

Doctorat - UdeM

matthew.riemer@mila.quebec

Alexis Roger

Maîtrise recherche - UdeM

alexis.roger@mila.quebec

Zahra Sheikhbahaee

Postdoctorat - UdeM

Superviseur⋅e principal⋅e :

zahra.sheikhbahaee@mila.quebec

Vaibhav Singh

Doctorat - Concordia

Superviseur⋅e principal⋅e :

vaibhav.singh@mila.quebec

Gopeshh Subbaraj

Doctorat - UdeM

gopeshh.subbaraj@mila.quebec

Doctorat - UdeM

Co-superviseur⋅e :

benjamin.therien@mila.quebec

Pascal Tikeng Notsawo

Doctorat - UdeM

Co-superviseur⋅e :

pascal.tikeng@mila.quebec

Maîtrise recherche - UdeM

tejas.vaidhya@mila.quebec

Dave Whipps

Maîtrise recherche - UdeM

whippsda@mila.quebec

andrew.williams@mila.quebec

Andrew Williams

Doctorat - UdeM

daria.yasafova@mila.quebec

Daria Yasafova

Stagiaire de recherche - Technical University of Munich

Omar Younis

Stagiaire de recherche - UdeM

omar.younis@mila.quebec

Publications

A Survey on Compositional Generalization in Applications

Baihan Lin

Djallel Bouneffouf

2023-02-02

ArXiv (prépublication)

Broken Neural Scaling Laws

Ethan Caballero

Kshitij Gupta

David Scott Krueger

We present a smoothly broken power law functional form (that we refer to as a Broken Neural Scaling Law (BNSL)) that accurately models&extra… (voir plus)polates the scaling behaviors of deep neural networks (i.e. how the evaluation metric of interest varies as amount of compute used for training (or inference), number of model parameters, training dataset size, model input size, number of training steps, or upstream performance varies) for various architectures&for each of various tasks within a large&diverse set of upstream&downstream tasks, in zero-shot, prompted,&finetuned settings. This set includes large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, AI alignment, AI capabilities, robotics, out-of-distribution (OOD) generalization, continual learning, transfer learning, uncertainty estimation / calibration, OOD detection, adversarial robustness, distillation, sparsity, retrieval, quantization, pruning, fairness, molecules, computer programming/coding, math word problems,"emergent phase transitions", arithmetic, supervised learning, unsupervised/self-supervised learning,&reinforcement learning (single agent&multi-agent). When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Moreover, this functional form accurately models&extrapolates scaling behavior that other functional forms are incapable of expressing such as the nonmonotonic transitions present in the scaling behavior of phenomena such as double descent&the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic. Lastly, we use this functional form to glean insights about the limit of the predictability of scaling behavior. Code is available at https://github.com/ethancaballero/broken_neural_scaling_laws

2023-02-01

ICLR.cc/2023/Conference (poster)

openreview.net

AI Agents Learn to Trust

Ardavan S. Nobandegani

T. Shultz

2023-01-01

Annual Meeting of the Cognitive Science Society (publié)

dblp.uni-trier.de

GOKU-UI: Ubiquitous Inference through Attention and Multiple Shooting for Continuous-time Generative Models

Germán Abrevaya

Mahta Ramezanian-Panahi

Jean-Christophe Gagnon-Audet

Pablo Polosecki

Silvina Ponce Dawson

Guillermo Cecchi

Scientiﬁc Machine Learning (SciML) is a burgeoning ﬁeld that synergistically combines domain-aware and interpretable models with agnosti… (voir plus)c machine learning techniques. In this work, we introduce GOKU-UI, an evolution of the SciML generative model GOKU-nets. The GOKU-UI broadens the original model’s spectrum to incorporate other classes of differential equations, such as Stochastic Differential Equations (SDEs), and integrates a distributed, i.e. ubiquitous, inference through attention mechanisms and a novel multiple shooting training strategy in the latent space. These enhancements have led to a signiﬁcant increase in its performance in both reconstruction and forecast tasks, as demonstrated by our evaluation of simulated and empirical data. Speciﬁcally, GOKU-UI outperformed all baseline models on synthetic datasets even with a training set 32-fold smaller, underscoring its remarkable data efﬁciency. Furthermore, when applied to empirical human brain data, while incorporating stochastic Stuart-Landau

2023-01-01

arXiv.org (prépublication)

Gradient Masked Averaging for Federated Learning

Irene Tenison

Sai Aravind Sreeramadas

Vaikkunth Mugunthan

Edouard Oyallon

Federated learning (FL) is an emerging paradigm that permits a large number of clients with heterogeneous data to coordinate learning of a u… (voir plus)nified global model without the need to share data amongst each other. A major challenge in federated learning is the heterogeneity of data across client, which can degrade the performance of standard FL algorithms. Standard FL algorithms involve averaging of model parameters or gradient updates to approximate the global model at the server. However, we argue that in heterogeneous settings, averaging can result in information loss and lead to poor generalization due to the bias induced by dominant client gradients. We hypothesize that to generalize better across non-i.i.d datasets, the algorithms should focus on learning the invariant mechanism that is constant while ignoring spurious mechanisms that differ across clients. Inspired from recent works in Out-of-Distribution generalization, we propose a gradient masked averaging approach for FL as an alternative to the standard averaging of client updates. This aggregation technique for client updates can be adapted as a drop-in replacement in most existing federated algorithms. We perform extensive experiments on multiple FL algorithms with in-distribution, real-world, feature-skewed out-of-distribution, and quantity imbalanced datasets and show that it provides consistent improvements, particularly in the case of heterogeneous clients.

2023-01-01

Trans. Mach. Learn. Res. (publié)

openreview.net

Lag-Llama: Towards Foundation Models for Time Series Forecasting

Kashif Rasul

Arjun Ashok

Andrew Robert Williams

Arian Khorasani

George Adamopoulos

Rishika Bhagwatkar

Marin Biloš

Hena Ghonia

N. Hassen

Anderson Schneider

Sahil Garg

Alexandre Drouin

Nicolas Chapados

Yuriy Nevmyvaka

Aiming to build foundation models for time-series forecasting and study their scaling behavior, we present here our work-in-progress on Lag-… (voir plus)Llama , a general-purpose univariate probabilistic time-series forecasting model trained on a large collection of time-series data. The model shows good zero-shot prediction capabilities on unseen “out-of-distribution” time-series datasets, outperforming supervised baselines. We use smoothly broken power-laws [7] to fit and predict model scaling behavior. The open source code is made available at https://github

2023-01-01

arXiv.org (prépublication)

Towards Continual Reinforcement Learning: A Review and Perspectives

Khimya Khetarpal

Matthew D Riemer

Doina Precup

2022-12-22

Journal of Artificial Intelligence Research (publié)

Continual Learning with Foundation Models: An Empirical Study of Latent Replay

Oleksiy Ostapenko

Timothee LESORT

Pau Rodriguez

Md Rifat Arefin

Arthur Douillard

Laurent Charlin

Rapid development of large-scale pre-training has resulted in foundation models that can act as effective feature extractors on a variety of… (voir plus) downstream tasks and domains. Motivated by this, we study the efficacy of pre-trained vision models as a foundation for downstream continual learning (CL) scenarios. Our goal is twofold. First, we want to understand the compute-accuracy trade-off between CL in the raw-data space and in the latent space of pre-trained encoders. Second, we investigate how the characteristics of the encoder, the pre-training algorithm and data, as well as of the resulting latent space affect CL performance. For this, we compare the efficacy of various pre-trained models in large-scale benchmarking scenarios with a vanilla replay setting applied in the latent and in the raw-data space. Notably, this study shows how transfer, forgetting, task similarity and learning are dependent on the input data characteristics and not necessarily on the CL algorithms. First, we show that under some circumstances reasonable CL performance can readily be achieved with a non-parametric classifier at negligible compute. We then show how models pre-trained on broader data result in better performance for various replay sizes. We explain this with representational similarity and transfer properties of these representations. Finally, we show the effectiveness of self-supervised pre-training for downstream domains that are out-of-distribution as compared to the pre-training domain. We point out and validate several research directions that can further increase the efficacy of latent CL including representation ensembling. The diverse set of datasets used in this study can serve as a compute-efficient playground for further CL research. We will publish the code.

2022-11-28

Proceedings of The 1st Conference on Lifelong Learning Agents (publié)

proceedings.mlr.press

APP: Anytime Progressive Pruning

Diganta Misra

Bharat Runwal

Tianlong Chen

Zhangyang Wang

With the latest advances in deep learning, several methods have been investigated for optimal learning settings in scenarios where the data … (voir plus)stream is continuous over time. However, training sparse networks in such settings has often been overlooked. In this paper, we explore the problem of training a neural network with a target sparsity in a particular case of online learning: the anytime learning at macroscale paradigm (ALMA). We propose a novel way of progressive pruning, referred to as \textit{Anytime Progressive Pruning} (APP); the proposed approach significantly outperforms the baseline dense and Anytime OSP models across multiple architectures and datasets under short, moderate, and long-sequence training. Our method, for example, shows an improvement in accuracy of

2022-11-17

ACML.org/2022/Workshop/CLL (publié)

openreview.net

Knowledge Distillation for Federated Learning: a Practical Guide

Alessio Mora

Irene Tenison

Paolo Bellavista

Federated Learning (FL) enables the training of Deep Learning models without centrally collecting possibly sensitive raw data. This paves th… (voir plus)e way for stronger privacy guarantees when building predictive models. The most used algorithms for FL are parameter-averaging based schemes (e.g., Federated Averaging) that, however, have well known limits: (i) Clients must implement the same model architecture; (ii) Transmitting model weights and model updates implies high communication cost, which scales up with the number of model parameters; (iii) In presence of non-IID data distributions, parameter-averaging aggregation schemes perform poorly due to client model drifts. Federated adaptations of regular Knowledge Distillation (KD) can solve and/or mitigate the weaknesses of parameter-averaging FL algorithms while possibly introducing other trade-offs. In this article, we provide a review of KD-based algorithms tailored for specific FL issues.

2022-11-09

ArXiv (prépublication)

Aligning MAGMA by Few-Shot Learning and Finetuning

Jean-Charles Layoun

Alexis Roger

2022-10-18

ArXiv (prépublication)