Publications

Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning
Aniket Didolkar
Kshitij Gupta
Anirudh Goyal
Nitesh B. Gundavarapu
Alex Lamb
Nan Rosemary Ke
Yoshua Bengio
Recurrent neural networks have a strong inductive bias towards learning temporally compressed representations, as the entire history of a se… (see more)quence is represented by a single vector. By contrast, Transformers have little inductive bias towards learning temporally compressed representations, as they allow for attention over all previously computed elements in a sequence. Having a more compressed representation of a sequence may be beneficial for generalization, as a high-level representation may be more easily re-used and re-purposed and will contain fewer irrelevant details. At the same time, excessive compression of representations comes at the cost of expressiveness. We propose a solution which divides computation into two streams. A slow stream that is recurrent in nature aims to learn a specialized and compressed representation, by forcing chunks of
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net
Trajectory balance: Improved credit assignment in GFlowNets
Nikolay Malkin
Moksh Jain
Emmanuel Bengio
Chen Sun
Yoshua Bengio
Generative flow networks (GFlowNets) are a method for learning a stochastic policy for generating compositional objects, such as graphs or s… (see more)trings, from a given unnormalized density by sequences of actions, where many possible action sequences may lead to the same object. We find previously proposed learning objectives for GFlowNets, flow matching and detailed balance, which are analogous to temporal difference learning, to be prone to inefficient credit propagation across long action sequences. We thus propose a new learning objective for GFlowNets, trajectory balance, as a more efficient alternative to previously used objectives. We prove that any global minimizer of the trajectory balance objective can define a policy that samples exactly from the target distribution. In experiments on four distinct domains, we empirically demonstrate the benefits of the trajectory balance objective for GFlowNet convergence, diversity of generated samples, and robustness to long action sequences and large action spaces.
2022-11-28
Neural Information Processing Systems (Accept)
doi.org
openreview.net
Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions
Kiwon Lee
Andrew N. Cheng
Courtney Paquette
Elliot Paquette
We analyze the dynamics of large batch stochastic gradient descent with momentum (SGD+M) on the least squares problem when both the number o… (see more)f samples and dimensions are large. In this setting, we show that the dynamics of SGD+M converge to a deterministic discrete Volterra equation as dimension increases, which we analyze. We identify a stability measurement, the implicit conditioning ratio (ICR), which regulates the ability of SGD+M to accelerate the algorithm. When the batch size exceeds this ICR, SGD+M converges linearly at a rate of
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net
Weakly Supervised Representation Learning with Sparse Perturbations
Kartik Ahuja
Jason Hartford
Yoshua Bengio
The theory of representation learning aims to build methods that provably invert the data generating process with minimal domain knowledge o… (see more)r any source of supervision. Most prior approaches require strong distributional assumptions on the latent variables and weak supervision (auxiliary information such as timestamps) to provide provable identification guarantees. In this work, we show that if one has weak supervision from observations generated by sparse perturbations of the latent variables--e.g. images in a reinforcement learning environment where actions move individual sprites--identification is achievable under unknown continuous latent distributions. We show that if the perturbations are applied only on mutually exclusive blocks of latents, we identify the latents up to those blocks. We also show that if these perturbation blocks overlap, we identify latents up to the smallest blocks shared across perturbations. Consequently, if there are blocks that intersect in one latent variable only, then such latents are identified up to permutation and scaling. We propose a natural estimation procedure based on this theory and illustrate it on low-dimensional synthetic and image-based experiments.
2022-11-28
Neural Information Processing Systems (Accept)
doi.org
openreview.net
Continual Learning with Foundation Models: An Empirical Study of Latent Replay
Oleksiy Ostapenko
Timothee LESORT
Pau Rodríguez
Md Rifat Arefin
Arthur Douillard
Irina Rish
Laurent Charlin
Rapid development of large-scale pre-training has resulted in foundation models that can act as effective feature extractors on a variety of… (see more) downstream tasks and domains. Motivated by this, we study the efficacy of pre-trained vision models as a foundation for downstream continual learning (CL) scenarios. Our goal is twofold. First, we want to understand the compute-accuracy trade-off between CL in the raw-data space and in the latent space of pre-trained encoders. Second, we investigate how the characteristics of the encoder, the pre-training algorithm and data, as well as of the resulting latent space affect CL performance. For this, we compare the efficacy of various pre-trained models in large-scale benchmarking scenarios with a vanilla replay setting applied in the latent and in the raw-data space. Notably, this study shows how transfer, forgetting, task similarity and learning are dependent on the input data characteristics and not necessarily on the CL algorithms. First, we show that under some circumstances reasonable CL performance can readily be achieved with a non-parametric classifier at negligible compute. We then show how models pre-trained on broader data result in better performance for various replay sizes. We explain this with representational similarity and transfer properties of these representations. Finally, we show the effectiveness of self-supervised pre-training for downstream domains that are out-of-distribution as compared to the pre-training domain. We point out and validate several research directions that can further increase the efficacy of latent CL including representation ensembling. The diverse set of datasets used in this study can serve as a compute-efficient playground for further CL research. The codebase is available under https://github.com/oleksost/latent_CL.
2022-11-27
Conference on Lifelong Learning Agents (published)
doi.org
proceedings.mlr.press
Improving Meta-Learning Generalization with Activation-Based Early-Stopping
Simon Guiroy
Christopher Pal
Goncalo Mordido
A. Chandar
2022-11-27
Proceedings of The 1st Conference on Lifelong Learning Agents (published)
doi.org
proceedings.mlr.press
Overcoming Challenges in Leveraging GANs for Few-Shot Data Augmentation
Christopher Beckham
Issam Laradji
Pau Rodríguez
David Vázquez
Derek Nowrouzezahrai
Christopher Pal
In this paper, we explore the use of GAN-based few-shot data augmentation as a method to improve few-shot classification performance. We per… (see more)form an exploration into how a GAN can be fine-tuned for such a task (one of which is in a class-incremental manner), as well as a rigorous empirical investigation into how well these models can perform to improve few-shot classification. We identify issues related to the difficulty of training such generative models under a purely supervised regime with very few examples, as well as issues regarding the evaluation protocols of existing works. We also find that in this regime, classification accuracy is highly sensitive to how the classes of the dataset are randomly split. Therefore, we propose a semi-supervised fine-tuning approach as a more pragmatic way forward to address these problems.
2022-11-27
Proceedings of The 1st Conference on Lifelong Learning Agents (published)
doi.org
proceedings.mlr.press
Pitfalls of Conditional Batch Normalization for Contextual Multi-Modal Learning
Ivaxi Sheth
Aamer Abdul Rahman
Mohammad Havaei
S Ebrahimi Kahou
Humans have perfected the art of learning from multiple modalities through sensory organs. Despite their impressive predictive performance o… (see more)n a single modality, neural networks cannot reach human level accuracy with respect to multiple modalities. This is a particularly challenging task due to variations in the structure of respective modalities. Conditional Batch Normalization (CBN) is a popular method that was proposed to learn contextual features to aid deep learning tasks. This technique uses auxiliary data to improve representational power by learning affine transformations for convolutional neural networks. Despite the boost in performance observed by using CBN layers, our work reveals that the visual features learned by introducing auxiliary data via CBN deteriorates. We perform comprehensive experiments to evaluate the brittleness of CBN networks to various datasets, suggesting that learning from visual features alone could often be superior for generalization. We evaluate CBN models on natural images for bird classification and histology images for cancer type classification. We observe that the CBN network learns close to no visual features on the bird classification dataset and partial visual features on the histology dataset. Our extensive experiments reveal that CBN may encourage shortcut learning between the auxiliary data and labels.
2022-11-27
ArXiv (preprint)
doi.org
arxiv.org
Shimming toolbox: An open‐source software toolbox for <scp>B0</scp> and <scp>B1</scp> shimming in MRI
Alexandre D'Astous
Gaspard Cereza
Daniel Papp
Kyle M. Gilbert
Jason P. Stockmann
Eva Alonso‐Ortiz
Julien Cohen‐Adad
Introduce Shimming Toolbox ( https://shimming‐toolbox.org), an open‐source software package for prototyping new methods and performing s… (see more)tatic, dynamic, and real‐time B0 shimming as well as B1 shimming experiments. Shimming Toolbox features various field mapping techniques, manual and automatic masking for the brain and spinal cord, B0 and B1 shimming capabilities accessible through a user‐friendly graphical user interface. Validation of Shimming Toolbox was demonstrated in three scenarios: (i) B0 dynamic shimming in the brain at 7T using custom AC/DC coils, (ii) B0 real‐time shimming in the spinal cord at 3T, and (iii) B1 static shimming in the spinal cord at 7T. Shimming Toolbox provides an open‐source platform where researchers can collaborate, prototype and conveniently test B0 and B1 shimming experiments. Future versions will include additional field map preprocessing techniques, optimization algorithms, and compatibility across multiple MRI manufacturers.
2022-11-27
Magnetic Resonance in Medicine (unknown)
doi.org
Receptive Field Refinement for Convolutional Neural Networks Reliably Improves Predictive Performance
Mats Leon Richter
Christopher Pal
Minimal changes to neural architectures (e.g. changing a single hyperparameter in a key layer), can lead to significant gains in predictive … (see more)performance in Convolutional Neural Networks (CNNs). In this work, we present a new approach to receptive field analysis that can yield these types of theoretical and empirical performance gains across twenty well-known CNN architectures examined in our experiments. By further developing and formalizing the analysis of receptive field expansion in convolutional neural networks, we can predict unproductive layers in an automated manner before ever training a model. This allows us to optimize the parameter-efficiency of a given architecture at low cost. Our method is computationally simple and can be done in an automated manner or even manually with minimal effort for most common architectures. We demonstrate the effectiveness of this approach by increasing parameter efficiency across past and current top-performing CNN-architectures. Specifically, our approach is able to improve ImageNet1K performance across a wide range of well-known, state-of-the-art (SOTA) model classes, including: VGG Nets, MobileNetV1, MobileNetV3, NASNet A (mobile), MnasNet, EfficientNet, and ConvNeXt - leading to a new SOTA result for each model class.
2022-11-25
ArXiv (preprint)
doi.org
arxiv.org
Applied artificial intelligence in healthcare: Listening to the winds of change in a post-COVID-19 world
Arash Shaban-Nejad
Martin Michalowski
Simone Bianco
John S. Brownstein
David L Buckeridge
Robert L Davis
2022-11-24
Experimental Biology and Medicine (published)
doi.org
Taxonomy of Benchmarks in Graph Representation Learning
Renming Liu
Semih Cantürk
Frederik Wenkel
Sarah McGuire
Xinyi Wang
Anna Little
Leslie O’Bray
Michael Perlmutter
Bastian Rieck
Matthew Hirn
Guy Wolf
Ladislav Rampasek
Graph Neural Networks (GNNs) extend the success of neural networks to graph-structured data by accounting for their intrinsic geometry. Whil… (see more)e extensive research has been done on developing GNN models with superior performance according to a collection of graph representation learning benchmarks, it is currently not well understood what aspects of a given model are probed by them. For example, to what extent do they test the ability of a model to leverage graph structure vs. node features? Here, we develop a principled approach to taxonomize benchmarking datasets according to a
2022-11-23
Learning on Graphs Conference (oral)
doi.org
proceedings.mlr.press