Publications

Effect of diversity in Meta-Learning
Ramnath Kumar
Tristan Deleu
Yoshua Bengio
Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples. Recent studies show that task … (voir plus)distribution plays a vital role in the performance of the model. Conventional wisdom is that task diversity should improve the performance of meta-learning. In this work, we find evidence to the contrary; we study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms. For this experiment, we train on two datasets - Omniglot and miniImageNet and with three broad classes of meta-learning models - Metric-based (i.e., Protonet, Matching Networks), Optimization-based (i.e., MAML, Reptile, and MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs). Our experiments demonstrate that the effect of task diversity on all these algorithms follows a similar trend, and task diversity does not seem to offer any benefits to the learning of the model. Furthermore, we also demonstrate that even a handful of tasks, repeated over multiple batches, would be sufficient to achieve a performance similar to uniform sampling and draws into question the need for additional tasks to create better models.
2021-12-09
NeurIPS.cc/2021/Workshop/MetaLearn (poster)
openreview.net
openreview.net
Few Shot Image Generation via Implicit Autoencoding of Support Sets
Andy Huang
Kuan-Chieh Wang
Guillaume Rabusseau
Alireza Makhzani
Recent generative models such as generative adversarial networks have achieved remarkable success in generating realistic images, but they r… (voir plus)equire large training datasets and computational resources. The goal of few-shot image generation is to learn the distribution of a new dataset from only a handful of examples by transferring knowledge learned from structurally similar datasets. Towards achieving this goal, we propose the “Implicit Support Set Autoencoder” (ISSA) that adversarially learns the relationship across datasets using an unsupervised dataset representation, while the distribution of each individual dataset is learned using implicit distributions. Given a few examples from a new dataset, ISSA can generate new samples by inferring the representation of the underlying distribution using a single forward pass. We showcase significant gains from our method on generating high quality and diverse images for unseen classes in the Omniglot and CelebA datasets in few-shot image generation settings.
2021-12-09
NeurIPS.cc/2021/Workshop/MetaLearn (poster)
openreview.net
openreview.net
Maternal chemosignals enhance infant-adult brain-to-brain synchrony
Yaara Endevelt-Shapira
Amir Djalovski
Guillaume Dumas
Ruth Feldman
2021-12-09
Science Advances (publié)
doi.org
Endocytic proteins with prion-like domains form viscoelastic condensates that enable membrane remodeling
Louis-Philippe Bergeron-Sandoval
Sandeep Kumar
Hossein Khadivi Heris
Catherine L. A. Chang
Caitlin E. Cornell
Sarah L. Keller
Paul François
Adam G. Hendricks
Allen J. Ehrlicher
Rohit V. Pappu
Stephen W. Michnick
The uptake of molecules into cells, known as endocytosis, requires membrane invagination and the formation of vesicles. A version of endocyt… (voir plus)osis that is independent of actin polymerization is aided by the assembly of membraneless biomolecular condensates at the site of membrane invagination. Here, we show that endocytic condensates are viscoelastic bodies that concentrate key proteins with prion-like domains to enable membrane remodeling. A distinct molecular grammar, namely the preference for glutamine versus asparagine residues, underlies the cohesive interactions that give rise to endocytic condensates. We incorporate material properties inferred using active rheology into a mechanical model to explain how cohesive interactions within condensates and interfacial tensions among condensates, membranes, and the cytosol can drive membrane invagination to initiate endocyosis.
2021-12-08
Proceedings of the National Academy of Sciences of the United States of America (publié)
doi.org
Adversarial Feature Desensitization
Pouya Bashivan
Reza Bayat
Adam Ibrahim
Kartik Ahuja
Mojtaba Faramarzi
Touraj Laleh
Blake Richards
Irina Rish
Neural networks are known to be vulnerable to adversarial attacks -- slight but carefully constructed perturbations of the inputs which can … (voir plus)drastically impair the network's performance. Many defense methods have been proposed for improving robustness of deep networks by training them on adversarially perturbed inputs. However, these models often remain vulnerable to new types of attacks not seen during training, and even to slightly stronger versions of previously seen attacks. In this work, we propose a novel approach to adversarial robustness, which builds upon the insights from the domain adaptation field. Our method, called Adversarial Feature Desensitization (AFD), aims at learning features that are invariant towards adversarial perturbations of the inputs. This is achieved through a game where we learn features that are both predictive and robust (insensitive to adversarial attacks), i.e. cannot be used to discriminate between natural and adversarial data. Empirical results on several benchmarks demonstrate the effectiveness of the proposed approach against a wide range of attack types and attack strengths. Our code is available at https://github.com/BashivanLab/afd.
2021-12-06
Conference on Neural Information Processing Systems (Accept (Poster))
doi.org
openreview.net
A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning
Mingde Zhao
Zhen Liu
Sitao Luan
Shuyuan Zhang
Doina Precup
Yoshua Bengio
We present an end-to-end, model-based deep reinforcement learning agent which dynamically attends to relevant parts of its state during plan… (voir plus)ning. The agent uses a bottleneck mechanism over a set-based representation to force the number of entities to which the agent attends at each planning step to be small. In experiments, we investigate the bottleneck mechanism with several sets of customized environments featuring different challenges. We consistently observe that the design allows the planning agents to generalize their learned task-solving abilities in compatible unseen environments by attending to the relevant objects, leading to better out-of-distribution generalization performance.
2021-12-06
Neural Information Processing Systems (Accept (Poster))
doi.org
openreview.net
Continual Learning via Local Module Composition
Oleksiy Ostapenko
Pau Rodríguez
Massimo Caccia
Laurent Charlin
Modularity is a compelling solution to continual learning (CL), the problem of modeling sequences of related tasks. Learning and then compos… (voir plus)ing modules to solve different tasks provides an abstraction to address the principal challenges of CL including catastrophic forgetting, backward and forward transfer across tasks, and sub-linear model growth. We introduce local module composition (LMC), an approach to modular CL where each module is provided a local structural component that estimates a module's relevance to the input. Dynamic module composition is performed layer-wise based on local relevance scores. We demonstrate that agnosticity to task identities (IDs) arises from (local) structural learning that is module-specific as opposed to the task- and/or model-specific as in previous works, making LMC applicable to more CL settings compared to previous works. In addition, LMC also tracks statistics about the input distribution and adds new modules when outlier samples are detected. In the first set of experiments, LMC performs favorably compared to existing methods on the recent Continual Transfer-learning Benchmark without requiring task identities. In another study, we show that the locality of structural learning allows LMC to interpolate to related but unseen tasks (OOD), as well as to compose modular networks trained independently on different task sequences into a third modular network without any fine-tuning. Finally, in search for limitations of LMC we study it on more challenging sequences of 30 and 100 tasks, demonstrating that local module selection becomes much more challenging in presence of a large number of candidate modules. In this setting best performing LMC spawns much fewer modules compared to an oracle based baseline, however, it reaches a lower overall accuracy. The codebase is available under https://github.com/oleksost/LMC.
2021-12-06
Conference on Neural Information Processing Systems (Accept (Poster))
doi.org
openreview.net
Curating the Twitter Election Integrity Datasets for Better Online Troll Characterization
Albert M. Orozco Camacho
Reihaneh Rabbany
In modern days, social media platforms provide accessible channels for the inter-1 action and immediate reflection of the most important ev… (voir plus)ents happening around 2 the world. In this paper, we, firstly, present a curated set of datasets whose origin 3 stem from the Twitter’s Information Operations 1 efforts. More notably, these 4 accounts, which have been already suspended, provide a notion of how state-backed 5 human trolls operate. 6 Secondly, we present detailed analyses of how these behaviours vary over time, 7 and motivate its use and abstraction in the context of deep representation learning: 8 for instance, to learn and, potentially track, troll behaviour. We present baselines 9 for such tasks and highlight the differences there may exist within the literature. 10 Finally, we utilize the representations learned for behaviour prediction to classify 11 trolls from "real" users, using a sample of non-suspended active accounts. 12
2021-12-06
LatinX in AI at Neural Information Processing Systems Conference 2021 (publié)
doi.org
openreview.net
Deep Reinforcement Learning at the Edge of the Statistical Precipice
Rishabh Agarwal
Max Schwarzer
Pablo Samuel Castro
Aaron Courville
Marc G. Bellemare
Deep reinforcement learning (RL) algorithms are predominantly evaluated by comparing their relative performance on a large suite of tasks. M… (voir plus)ost published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally-demanding benchmarks has led to the practice of evaluating only a small number of runs per task, exacerbating the statistical uncertainty in point estimates. In this paper, we argue that reliable evaluation in the few run deep RL regime cannot ignore the uncertainty in results without running the risk of slowing down progress in the field. We illustrate this point using a case study on the Atari 100k benchmark, where we find substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis. With the aim of increasing the field's confidence in reported results with a handful of runs, we advocate for reporting interval estimates of aggregate performance and propose performance profiles to account for the variability in results, as well as present more robust and efficient aggregate metrics, such as interquartile mean scores, to achieve small uncertainty in results. Using such statistical tools, we scrutinize performance evaluations of existing algorithms on other widely used RL benchmarks including the ALE, Procgen, and the DeepMind Control Suite, again revealing discrepancies in prior comparisons. Our findings call for a change in how we evaluate performance in deep RL, for which we present a more rigorous evaluation methodology, accompanied with an open-source library rliable, to prevent unreliable results from stagnating the field.
2021-12-06
Neural Information Processing Systems (Accept (Oral))
doi.org
openreview.net
Discrete-Valued Neural Communication in Structured Architectures Enhances Generalization
Dianbo Liu
Alex Lamb
Kenji Kawaguchi
Anirudh Goyal
Chen Sun
Michael C. Mozer
Yoshua Bengio
Deep learning has advanced from fully connected architectures to structured models organized into components, e.g., the transformer composed… (voir plus) of positional elements, modular architectures divided into slots, and graph neural nets made up of nodes. In structured models, an interesting question is how to conduct dynamic and possibly sparse communication among the separate components. Here, we explore the hypothesis that restricting the transmitted information among components to discrete representations is a beneficial bottleneck. The motivating intuition is human language in which communication occurs through discrete symbols. Even though individuals have different understandings of what a "cat" is based on their specific experiences, the shared discrete token makes it possible for communication among individuals to be unimpeded by individual differences in internal representation. To discretize the values of concepts dynamically communicated among specialist components, we extend the quantization mechanism from the Vector-Quantized Variational Autoencoder to multi-headed discretization with shared codebooks and use it for discrete-valued neural communication (DVNC). Our experiments show that DVNC substantially improves systematic generalization in a variety of architectures -- transformers, modular architectures, and graph neural networks. We also show that the DVNC is robust to the choice of hyperparameters, making the method very useful in practice. Moreover, we establish a theoretical justification of our discretization process, proving that it has the ability to increase noise robustness and reduce the underlying dimensionality of the model.
2021-12-06
Advances in Neural Information Processing Systems (Accept (Poster))
doi.org
openreview.net
Dynamic Inference with Neural Interpreters
Nasim Rahaman
Muhammad Waleed Gondal
Shruti Joshi
Peter Gehler
Yoshua Bengio
Francesco Locatello
Bernhard Schölkopf
Modern neural network architectures can leverage large amounts of data to generalize well within the training distribution. However, they ar… (voir plus)e less capable of systematic generalization to data drawn from unseen but related distributions, a feat that is hypothesized to require compositional reasoning and reuse of knowledge. In this work, we present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules, which we call \emph{functions}. Inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. The proposed architecture can flexibly compose computation along width and depth, and lends itself well to capacity extension after training. To demonstrate the versatility of Neural Interpreters, we evaluate it in two distinct settings: image classification and visual abstract reasoning on Raven Progressive Matrices. In the former, we show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner. In the latter, we find that Neural Interpreters are competitive with respect to the state-of-the-art in terms of systematic generalization
2021-12-06
Conference on Neural Information Processing Systems (Accept (Poster))
doi.org
openreview.net
Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models
Courtney Paquette
Elliot Paquette
We analyze a class of stochastic gradient algorithms with momentum on a high-dimensional random least squares problem. Our framework, inspir… (voir plus)ed by random matrix theory, provides an exact (deterministic) characterization for the sequence of loss values produced by these algorithms which is expressed only in terms of the eigenvalues of the Hessian. This leads to simple expressions for nearly-optimal hyperparameters, a description of the limiting neighborhood, and average-case complexity. As a consequence, we show that (small-batch) stochastic heavy-ball momentum with a fixed momentum parameter provides no actual performance improvement over SGD when step sizes are adjusted correctly. For contrast, in the non-strongly convex setting, it is possible to get a large improvement over SGD using momentum. By introducing hyperparameters that depend on the number of samples, we propose a new algorithm sDANA (stochastic dimension adjusted Nesterov acceleration) which obtains an asymptotically optimal average-case complexity while remaining linearly convergent in the strongly convex setting without adjusting parameters.
2021-12-06
Conference on Neural Information Processing Systems (Accept (Poster))
doi.org
openreview.net