Eugene Belilovsky

pietro.cagnasso@mila.quebec

Xiaolong Huang

Master's Research - Concordia University

Co-supervisor :

xiaolong.huang@mila.quebec

Paul Janson

PhD - Concordia University

Co-supervisor :

paul.janson@mila.quebec

Gwen Legate

PhD - Concordia University

Co-supervisor :

gwendolyne.legate@mila.quebec

Paria Mehrbod

Master's Research - Concordia University

Co-supervisor :

paria.mehrbod@mila.quebec

Abhinav Moudgil

PhD - Concordia University

abhinav.moudgil@mila.quebec

Adel Nabli

PhD - Concordia University

adel.nabli@mila.quebec

Geraldin Nanfack

Postdoctorate - Concordia University

Co-supervisor :

geraldin.nanfack@mila.quebec

tirathraj.ramburn@mila.quebec

Albert Orozco Camacho

PhD - Concordia University

Co-supervisor :

Master's Research - Université de Montréal

Ammar Sherif

PhD - Concordia University

Co-supervisor :

ammar.sherif@mila.quebec

Vaibhav Singh

PhD - Concordia University

Co-supervisor :

vaibhav.singh@mila.quebec

Benjamin Therien

PhD - Université de Montréal

Principal supervisor :

benjamin.therien@mila.quebec

Research Intern - Concordia University

junior-cedric.tonga@mila.quebec

Zhang Yanlei

Collaborating Alumni - Université de Montréal

Principal supervisor :

yanlei.zhang@mila.quebec

amirhossein.zamani@mila.quebec

Amirhossein Zamani

PhD - Concordia University

Co-supervisor :

Blake Richards

Publications

Simulated Annealing in Early Layers Leads to Better Generalization

Amir M. Sarfi

Zahra Karimpour

Muawiz Chaudhary

Nasir M. Khalid

Mirco Ravanelli

Sudhir Mudur

Recently, a number of iterative learning methods have been introduced to improve generalization. These typically rely on training for longer… (see more) periods of time in exchange for improved generalization. LLF (later-layer-forgetting) is a state-of-the-art method in this category. It strengthens learning in early layers by periodically re-initializing the last few layers of the network. Our principal innovation in this work is to use Simulated annealing in EArly Layers (SEAL) of the network in place of re-initialization of later layers. Essentially, later layers go through the normal gradient descent process, while the early layers go through short stints of gradient ascent followed by gradient descent. Extensive experiments on the popular Tiny-ImageNet dataset benchmark and a series of transfer learning and few-shot learning tasks show that we outperform LLF by a significant margin. We further show that, compared to normal training, LLF features, although improving on the target task, degrade the transfer learning performance across all datasets we explored. In comparison, our method outperforms LLF across the same target datasets by a large margin. We also show that the prediction depth of our method is significantly lower than that of LLF and normal training, indicating on average better prediction performance.

2023-06-19

IEEE/CVF Conference on Computer Vision and Pattern Recognition (Accept)

Reliability of CKA as a Similarity Measure in Deep Learning

MohammadReza Davari

Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different w… (see more)ays. The Centered Kernel Alignment (CKA) similarity metric, particularly its linear variant, has recently become a popular approach and has been widely used to compare representations of a network's different layers, of architecturally similar networks trained differently, or of models with different architectures trained on the same data. A wide variety of conclusions about similarity and dissimilarity of these various representations have been made using CKA. In this work we present analysis that formally characterizes CKA sensitivity to a large class of simple transformations, which can naturally occur in the context of modern machine learning. This provides a concrete explanation of CKA sensitivity to outliers, which has been observed in past works, and to transformations that preserve the linear separability of the data, an important generalization attribute. We empirically investigate several weaknesses of the CKA similarity metric, demonstrating situations in which it gives unexpected or counter-intuitive results. Finally we study approaches for modifying representations to maintain functional behaviour while changing the CKA value. Our results illustrate that, in many cases, the CKA value can be easily manipulated without substantial changes to the functional behaviour of the models, and call for caution when leveraging activation alignment metrics.

2023-04-30

International Conference on Learning Representations (Accept: poster)

Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning

Gwen Legate

Lucas Caccia

In Federated Learning a global model is learned by aggregating model updates computed at a set of independent client nodes. To reduce commun… (see more)ication costs, multiple gradient steps are performed at each node prior to aggregation. A key challenge in this setting is data heterogeneity across clients resulting in differing local objectives. This can lead clients to overly minimize their own local objective consequently diverging from the global solution. We demonstrate that individual client models experience a catastrophic forgetting with respect to data from other clients and propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss. This approach shields classes outside a client’s label set from abrupt representation change and we empirically demonstrate it can alleviate client forgetting and provide consistent improvements to standard federated learning algorithms. Our method is particularly beneficial under the most challenging federated learning settings where data heterogeneity is high and client participation in each round is low.

2022-12-31

Conference on Lifelong Learning Agents (published)

proceedings.mlr.press

Imitation from Observation With Bootstrapped Contrastive Learning

Medric Sonwa

Johanna Hansen

2022-12-08

NeurIPS.cc/2022/Workshop/DeepRL (accepted)

CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

Nasir Mohammad Khalid

Tianhao Xie

Tiberiu Popa

We present a technique for zero-shot generation of a 3D model using only a target text prompt. Without any 3D supervision our method deforms… (see more) the control shape of a limit subdivided surface along with its texture map and normal map to obtain a 3D asset that corresponds to the input text prompt and can be easily deployed into games or modeling applications. We rely only on a pre-trained CLIP model that compares the input text prompt with differentiably rendered images of our 3D model. While previous works have focused on stylization or required training of generative models we perform optimization on mesh parameters directly to generate shape, texture or both. To constrain the optimization to produce plausible meshes and textures we introduce a number of techniques using image augmentations and the use of a pretrained prior that generates CLIP image embeddings given a text embedding.

2022-11-29

SIGGRAPH Asia 2022 Conference Papers (published)

Towards Scaling Difference Target Propagation by Learning Backprop Targets

The development of biologically-plausible learning algorithms is important for understanding learning in the brain, but most of them fail to… (see more) scale-up to real-world tasks, limiting their potential as explanations for learning by real brains. As such, it is important to explore learning algorithms that come with strong theoretical guarantees and can match the performance of backpropagation (BP) on complex tasks. One such algorithm is Difference Target Propagation (DTP), a biologically-plausible learning algorithm whose close relation with Gauss-Newton (GN) optimization has been recently established. However, the conditions under which this connection rigorously holds preclude layer-wise training of the feedback pathway synaptic weights (which is more biologically plausible). Moreover, good alignment between DTP weight updates and loss gradients is only loosely guaranteed and under very specific conditions for the architecture being trained. In this paper, we propose a novel feedback weight training scheme that ensures both that DTP approximates BP and that layer-wise feedback weight training can be restored without sacrificing any theoretical guarantees. Our theory is corroborated by experimental results and we report the best performance ever achieved by DTP on CIFAR-10 and ImageNet 32

2022-07-18

International Conference on Machine Learning (Accept for Short Presentation)

proceedings.mlr.press

Probing Representation Forgetting in Supervised and Unsupervised Continual Learning

MohammadReza Davari

Nader Asadi

Sudhir Mudur

Rahaf Aljundi

Continual Learning research typically focuses on tackling the phenomenon of catastrophic forgetting in neural networks. Catastrophic forgett… (see more)ing is associated with an abrupt loss of knowledge previously learned by a model when the task, or more broadly the data distribution, being trained on changes. In supervised learning problems this forgetting, resulting from a change in the model's representation, is typically measured or observed by evaluating the decrease in old task performance. However, a model's representation can change without losing knowledge about prior tasks. In this work we consider the concept of representation forgetting, observed by using the difference in performance of an optimal linear classifier before and after a new task is introduced. Using this tool we revisit a number of standard continual learning benchmarks and observe that, through this lens, model representations trained without any explicit control for forgetting often experience small representation forgetting and can sometimes be comparable to methods which explicitly control for forgetting, especially in longer task sequences. We also show that representation forgetting can lead to new insights on the effect of model capacity and loss function used in continual learning. Based on our results, we show that a simple yet competitive approach is to learn representations continually with standard supervised contrastive learning while constructing prototypes of class samples when queried on old samples.

2022-06-17

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (published)

Revisiting Learnable Affines for Batch Norm in Few-Shot Transfer Learning

Moslem Yazdanpanah

Aamer Abdul Rahman

Muawiz Sajjad Chaudhary

Christian Desrosiers

Mohammad Havaei

S Ebrahimi Kahou

Batch normalization is a staple of computer vision models, including those employed in few-shot learning. Batch nor-malization layers in con… (see more)volutional neural networks are composed of a normalization step, followed by a shift and scale of these normalized features applied via the per-channel trainable affine parameters

2022-06-17

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (published)

Parametric Scattering Networks

Shanel Gauthier

Benjamin Therien

Laurent Alsène-Racicot

Michael Eickenberg

The wavelet scattering transform creates geometric invariants and deformation stability. In multiple signal domains, it has been shown to yi… (see more)eld more discriminative representations compared to other non-learned representations and to outperform learned representations in certain tasks, particularly on limited labeled data and highly structured signals. The wavelet filters used in the scattering transform are typically selected to create a tight frame via a parameterized mother wavelet. In this work, we investigate whether this standard wavelet filterbank construction is optimal. Focusing on Morlet wavelets, we propose to learn the scales, orientations, and aspect ratios of the filters to produce problem-specific parameterizations of the scattering transform. We show that our learned versions of the scattering transform yield significant performance gains in small-sample classification settings over the standard scattering transform. Moreover, our empirical results suggest that traditional filterbank constructions may not always be necessary for scattering transforms to extract effective representations.

2022-05-31

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (published)

New Insights on Reducing Abrupt Representation Change in Online Continual Learning

Lucas Caccia

Tinne Tuytelaars

In the online continual learning paradigm, agents must learn from a changing distribution while respecting memory and compute constraints. E… (see more)xperience Replay (ER), where a small subset of past data is stored and replayed alongside new data, has emerged as a simple and effective learning strategy. In this work, we focus on the change in representations of observed data that arises when previously unobserved classes appear in the incoming data stream, and new classes must be distinguished from previous ones. We shed new light on this question by showing that applying ER causes the newly added classes' representations to overlap significantly with the previous classes, leading to highly disruptive parameter updates. Based on this empirical analysis, we propose a new method which mitigates this issue by shielding the learned representations from drastic adaptation to accommodate new classes. We show that using an asymmetric update rule pushes new classes to adapt to the older ones (rather than the reverse), which is more effective especially at task boundaries, where much of the forgetting typically occurs. Empirical results show significant gains over strong baselines on standard continual learning benchmarks

2022-04-24

International Conference on Learning Representations (Accept (Poster))

Local Learning with Neuron Groups

Adeetya Patel

Michael Eickenberg

2022-04-20

ICLR.cc/2022/Workshop/Cells2Societies (poster)

Generative Compositional Augmentations for Scene Graph Prediction

Boris Knyazev

Harm de Vries

Cătălina Cangea

Graham W. Taylor

Aaron Courville

Inferring objects and their relationships from an image in the form of a scene graph is useful in many applications at the intersection of v… (see more)ision and language. We consider a challenging problem of compositional generalization that emerges in this task due to a long tail data distribution. Current scene graph generation models are trained on a tiny fraction of the distribution corresponding to the most frequent compositions, e.g. . However, test images might contain zero- and few-shot compositions of objects and relationships, e.g. . Despite each of the object categories and the predicate (e.g. 'on') being frequent in the training data, the models often fail to properly understand such unseen or rare compositions. To improve generalization, it is natural to attempt increasing the diversity of the training distribution. However, in the graph domain this is non-trivial. To that end, we propose a method to synthesize rare yet plausible scene graphs by perturbing real ones. We then propose and empirically study a model based on conditional generative adversarial networks (GANs) that allows us to generate visual features of perturbed scene graphs and learn from them in a joint fashion. When evaluated on the Visual Genome dataset, our approach yields marginal, but consistent improvements in zero- and few-shot metrics. We analyze the limitations of our approach indicating promising directions for future research.

2021-09-30

2021 IEEE/CVF International Conference on Computer Vision (ICCV) (published)