Eugene Belilovsky

nicolas.bernier@mila.quebec

nader.asadi@mila.quebec

Maîtrise recherche - Concordia

mohammadreza.davari@mila.quebec

Reza Davari

Doctorat - Concordia

Stagiaire de recherche - Concordia

louis.fournier@mila.quebec

Maîtrise recherche - Concordia

alexander.fulleringer@mila.quebec

Paul Janson

Maîtrise recherche - Concordia

paul.janson@mila.quebec

Charles-Etienne Joseph

Maîtrise recherche - UdeM

Co-superviseur⋅e :

charles-etienne.joseph@mila.quebec

Zafir Khalid

Maîtrise recherche - Concordia

zafir.khaled@mila.quebec

Gwen Legate

Doctorat - Concordia

Co-superviseur⋅e :

gwendolyne.legate@mila.quebec

abhinav.moudgil@mila.quebec

Abhinav Moudgil

Doctorat - Concordia

Adel Nabli

Doctorat - Concordia

adel.nabli@mila.quebec

Geraldin Nanfack

Postdoctorat - Concordia

Co-superviseur⋅e :

geraldin.nanfack@mila.quebec

Albert Orozco Camacho

Doctorat - Concordia

Co-superviseur⋅e :

Maîtrise recherche - Concordia

Co-superviseur⋅e :

paria.mehrbod@mila.quebec

Donald Shenaj

Collaborateur·rice de recherche - Concordia

Co-superviseur⋅e :

donald.shenaj@mila.quebec

Doctorat - Concordia

Co-superviseur⋅e :

vaibhav.singh@mila.quebec

Benjamin Therien

Doctorat - UdeM

Superviseur⋅e principal⋅e :

benjamin.therien@mila.quebec

Collaborateur·rice de recherche - UdeM

Superviseur⋅e principal⋅e :

pedro.vianna@mila.quebec

Humza Wajid Hameed

Maîtrise recherche - Concordia

humza.wajid@mila.quebec

amirhossein.zamani@mila.quebec

AmirHossein Zamani

Doctorat - Concordia

Congshu Zou

Maîtrise recherche - Concordia

congshu.zou@mila.quebec

Publications

Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning

Nader Asadi

MohammadReza Davari

Sudhir Mudur

Rahaf Aljundi

In Continual learning (CL) balancing effective adaptation while combating catastrophic forgetting is a central challenge. Many of the recent… (voir plus) best-performing methods utilize various forms of prior task data, e.g. a replay buffer, to tackle the catastrophic forgetting problem. Having access to previous task data can be restrictive in many real-world scenarios, for example when task data is sensitive or proprietary. To overcome the necessity of using previous tasks' data, in this work, we start with strong representation learning methods that have been shown to be less prone to forgetting. We propose a holistic approach to jointly learn the representation and class prototypes while maintaining the relevance of old class prototypes and their embedded similarities. Specifically, samples are mapped to an embedding space where the representations are learned using a supervised contrastive loss. Class prototypes are evolved continually in the same latent space, enabling learning and prediction at any point. To continually adapt the prototypes without keeping any prior task data, we propose a novel distillation loss that constrains class prototypes to maintain relative similarities as compared to new task data. This method yields state-of-the-art performance in the task-incremental setting, outperforming methods relying on large amounts of data, and provides strong performance in the class-incremental setting without using any stored data points.

2023-01-01

ICML (publié)

Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning

Gwen Legate

Lucas Caccia

In Federated Learning a global model is learned by aggregating model updates computed at a set of independent client nodes. To reduce commun… (voir plus)ication costs, multiple gradient steps are performed at each node prior to aggregation. A key challenge in this setting is data heterogeneity across clients resulting in differing local objectives. This can lead clients to overly minimize their own local objective consequently diverging from the global solution. We demonstrate that individual client models experience a catastrophic forgetting with respect to data from other clients and propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss. This approach shields classes outside a client’s label set from abrupt representation change and we empirically demonstrate it can alleviate client forgetting and provide consistent improvements to standard federated learning algorithms. Our method is particularly beneficial under the most challenging federated learning settings where data heterogeneity is high and client participation in each round is low.

2023-01-01

CoLLAs (publié)

Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning

Gwen Legate

Lucas Caccia

In Federated Learning, a global model is learned by aggregating model updates computed at a set of independent client nodes, to reduce commu… (voir plus)nication costs multiple gradient steps are performed at each node prior to aggregation. A key challenge in this setting is data heterogeneity across clients resulting in differing local objectives which can lead clients to overly minimize their own local objective, diverging from the global solution. We demonstrate that individual client models experience a catastrophic forgetting with respect to data from other clients and propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss. This approach shields classes outside a client's label set from abrupt representation change and we empirically demonstrate it can alleviate client forgetting and provide consistent improvements to standard federated learning algorithms. Our method is particularly beneficial under the most challenging federated learning settings where data heterogeneity is high and client participation in each round is low.

2023-01-01

CoLLAs (publié)

Imitation from Observation With Bootstrapped Contrastive Learning

Medric Sonwa

Johanna Hansen

2022-12-09

NeurIPS.cc/2022/Workshop/DeepRL (inconnu)

CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

Nasir M. Khalid

Tianhao Xie

Tiberiu Popa

2022-11-30

SIGGRAPH Asia 2022 Conference Papers (publié)

Towards Scaling Difference Target Propagation by Learning Backprop Targets

Maxence Ernoult

Fabrice Normandin

Abhinav Moudgil

Sean Spinney

The development of biologically-plausible learning algorithms is important for understanding learning in the brain, but most of them fail to… (voir plus) scale-up to real-world tasks, limiting their potential as explanations for learning by real brains. As such, it is important to explore learning algorithms that come with strong theoretical guarantees and can match the performance of backpropagation (BP) on complex tasks. One such algorithm is Difference Target Propagation (DTP), a biologically-plausible learning algorithm whose close relation with Gauss-Newton (GN) optimization has been recently established. However, the conditions under which this connection rigorously holds preclude layer-wise training of the feedback pathway synaptic weights (which is more biologically plausible). Moreover, good alignment between DTP weight updates and loss gradients is only loosely guaranteed and under very specific conditions for the architecture being trained. In this paper, we propose a novel feedback weight training scheme that ensures both that DTP approximates BP and that layer-wise feedback weight training can be restored without sacrificing any theoretical guarantees. Our theory is corroborated by experimental results and we report the best performance ever achieved by DTP on CIFAR-10 and ImageNet 32

2022-06-28

Proceedings of the 39th International Conference on Machine Learning (publié)

proceedings.mlr.press

Parametric Scattering Networks

Shanel Gauthier

Benjamin Thérien

Laurent Alséne-Racicot

Muawiz Chaudhary

Michael Eickenberg

The wavelet scattering transform creates geometric in-variants and deformation stability. In multiple signal do-mains, it has been shown to … (voir plus)yield more discriminative rep-resentations compared to other non-learned representations and to outperform learned representations in certain tasks, particularly on limited labeled data and highly structured signals. The wavelet filters used in the scattering trans-form are typically selected to create a tight frame via a pa-rameterized mother wavelet. In this work, we investigate whether this standard wavelet filterbank construction is op-timal. Focusing on Morlet wavelets, we propose to learn the scales, orientations, and aspect ratios of the filters to produce problem-specific parameterizations of the scattering transform. We show that our learned versions of the scattering transform yield significant performance gains in small-sample classification settings over the standard scat-tering transform. Moreover, our empirical results suggest that traditional filterbank constructions may not always be necessary for scattering transforms to extract effective rep-resentations.

2022-06-18

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (publié)

Probing Representation Forgetting in Supervised and Unsupervised Continual Learning

MohammadReza Davari

Nader Asadi

Sudhir Mudur

Rahaf Aljundi

Continual Learning (CL) research typically focuses on tackling the phenomenon of catastrophic forgetting in neural networks. Catastrophic fo… (voir plus)rgetting is associated with an abrupt loss of knowledge previously learned by a model when the task, or more broadly the data distribution, being trained on changes. In supervised learning problems this forgetting, resulting from a change in the model's representation, is typically measured or observed by evaluating the decrease in old task performance. However, a model's representation can change without losing knowledge about prior tasks. In this work we consider the concept of representation forgetting, observed by using the difference in performance of an optimal linear classifier before and after a new task is introduced. Using this tool we revisit a number of standard continual learning benchmarks and observe that, through this lens, model representations trained without any explicit control for forgetting often experience small representation forgetting and can sometimes be comparable to methods which explicitly control for forgetting, especially in longer task sequences. We also show that representation forgetting can lead to new insights on the effect of model capacity and loss function used in continual learning. Based on our results, we show that a simple yet competitive approach is to learn representations continually with standard supervised contrastive learning while constructing prototypes of class samples when queried on old samples.11The code to reproduce our results is publicly available at: https://github.com/rezazzr/Probing-Representation-Forgetting

2022-06-18

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (publié)

Revisiting Learnable Affines for Batch Norm in Few-Shot Transfer Learning

Moslem Yazdanpanah

Aamer Abdul Rahman

Muawiz Chaudhary

Christian Desrosiers

Mohammad Havaei

Samira Ebrahimi Kahou

Batch normalization is a staple of computer vision models, including those employed in few-shot learning. Batch nor-malization layers in con… (voir plus)volutional neural networks are composed of a normalization step, followed by a shift and scale of these normalized features applied via the per-channel trainable affine parameters

2022-06-18

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (publié)

Local Learning with Neuron Groups

Adeetya Patel

Michael Eickenberg

2022-04-21

ICLR.cc/2022/Workshop/Cells2Societies (poster)

CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

Nasir M. Khalid

Tianhao Xie

Tiberiu S. Popa

We present a technique for zero-shot generation of a 3D model using only a target text prompt. Without any 3D supervision our method deforms… (voir plus) the control shape of a limit subdivided surface along with its texture map and normal map to obtain a 3D asset that corresponds to the input text prompt and can be easily deployed into games or modeling applications. We rely only on a pre-trained CLIP model that compares the input text prompt with differentiably rendered images of our 3D model. While previous works have focused on stylization or required training of generative models we perform optimization on mesh parameters directly to generate shape, texture or both. To constrain the optimization to produce plausible meshes and textures we introduce a number of techniques using image augmentations and the use of a pretrained prior that generates CLIP image embeddings given a text embedding.

2022-03-24

ArXiv (preprint)

Probing Representation Forgetting in Supervised and Unsupervised Continual Learning

MohammadReza Davari

Nader Asadi

Sudhir Mudur

Rahaf Aljundi

2022-03-24

ArXiv (preprint)