Eugene Belilovsky

nicolas.bernier@mila.quebec

nader.asadi@mila.quebec

Maîtrise recherche - Concordia

mohammadreza.davari@mila.quebec

Reza Davari

Doctorat - Concordia

Stagiaire de recherche - Concordia

louis.fournier@mila.quebec

Maîtrise recherche - Concordia

alexander.fulleringer@mila.quebec

Paul Janson

Maîtrise recherche - Concordia

paul.janson@mila.quebec

Charles-Etienne Joseph

Maîtrise recherche - UdeM

Co-superviseur⋅e :

charles-etienne.joseph@mila.quebec

Zafir Khalid

Maîtrise recherche - Concordia

zafir.khaled@mila.quebec

Gwen Legate

Doctorat - Concordia

Co-superviseur⋅e :

gwendolyne.legate@mila.quebec

abhinav.moudgil@mila.quebec

Abhinav Moudgil

Doctorat - Concordia

Adel Nabli

Doctorat - Concordia

adel.nabli@mila.quebec

Geraldin Nanfack

Postdoctorat - Concordia

Co-superviseur⋅e :

geraldin.nanfack@mila.quebec

Albert Orozco Camacho

Doctorat - Concordia

Co-superviseur⋅e :

Maîtrise recherche - Concordia

Co-superviseur⋅e :

paria.mehrbod@mila.quebec

Donald Shenaj

Collaborateur·rice de recherche - Concordia

Co-superviseur⋅e :

donald.shenaj@mila.quebec

Doctorat - Concordia

Co-superviseur⋅e :

vaibhav.singh@mila.quebec

Benjamin Therien

Doctorat - UdeM

Superviseur⋅e principal⋅e :

benjamin.therien@mila.quebec

Collaborateur·rice de recherche - UdeM

Superviseur⋅e principal⋅e :

pedro.vianna@mila.quebec

Humza Wajid Hameed

Maîtrise recherche - Concordia

humza.wajid@mila.quebec

amirhossein.zamani@mila.quebec

AmirHossein Zamani

Doctorat - Concordia

Congshu Zou

Maîtrise recherche - Concordia

congshu.zou@mila.quebec

Publications

New Insights on Reducing Abrupt Representation Change in Online Continual Learning

Lucas Caccia

Rahaf Aljundi

Nader Asadi

Tinne Tuytelaars

Joelle Pineau

In the online continual learning paradigm, agents must learn from a changing distribution while respecting memory and compute constraints. E… (voir plus)xperience Replay (ER), where a small subset of past data is stored and replayed alongside new data, has emerged as a simple and effective learning strategy. In this work, we focus on the change in representations of observed data that arises when previously unobserved classes appear in the incoming data stream, and new classes must be distinguished from previous ones. We shed new light on this question by showing that applying ER causes the newly added classes’ representations to overlap significantly with the previous classes, leading to highly disruptive parameter updates. Based on this empirical analysis, we propose a new method which mitigates this issue by shielding the learned representations from drastic adaptation to accommodate new classes. We show that using an asymmetric update rule pushes new classes to adapt to the older ones (rather than the reverse), which is more effective especially at task boundaries, where much of the forgetting typically occurs. Empirical results show significant gains over strong baselines on standard continual learning benchmarks.

2022-01-28

ICLR.cc/2022/Conference (poster)

doi.org

openreview.net

Parametric Scattering Networks

Shanel Gauthier

Benjamin Th'erien

Laurent Alséne-Racicot

Michael Eickenberg

The wavelet scattering transform creates geometric in-variants and deformation stability. In multiple signal do-mains, it has been shown to … (voir plus)yield more discriminative rep-resentations compared to other non-learned representations and to outperform learned representations in certain tasks, particularly on limited labeled data and highly structured signals. The wavelet filters used in the scattering trans-form are typically selected to create a tight frame via a pa-rameterized mother wavelet. In this work, we investigate whether this standard wavelet filterbank construction is op-timal. Focusing on Morlet wavelets, we propose to learn the scales, orientations, and aspect ratios of the filters to produce problem-specific parameterizations of the scattering transform. We show that our learned versions of the scattering transform yield significant performance gains in small-sample classification settings over the standard scat-tering transform. Moreover, our empirical results suggest that traditional filterbank constructions may not always be necessary for scattering transforms to extract effective rep-resentations.

2021-07-20

ArXiv (preprint)

doi.org

arxiv.org

Online Continual Learning with Maximally Interfered Retrieval

Rahaf Aljundi

Lucas Caccia

Massimo Caccia

Min Lin

Laurent Charlin

Tinne Tuytelaars

Continual learning, the setting where a learning agent is faced with a never ending stream of data, continues to be a great challenge for mo… (voir plus)dern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks. These approaches typically rely on randomly selecting samples from the replay memory or from a generative model, which is suboptimal. In this work, we consider a controlled sampling of memories for replay. We retrieve the samples which are most interfered, i.e. whose prediction will be most negatively impacted by the foreseen parameters update. We show a formulation for this sampling criterion in both the generative replay and the experience replay setting, producing consistent gains in performance and greatly reduced forgetting. We release an implementation of our method at this https URL.

2019-08-11

ArXiv (prépublication)

arxiv.org

Learning Optimizers for Local SGD

Charles-Étienne Joseph

Benjamin Thérien

Abhinav Moudgil

Boris Knyazev