Publications

Combining Parameter-efficient Modules for Task-level Generalisation
Edoardo M. Ponti
Alessandro Sordoni
Yoshua Bengio
Siva Reddy
A modular design encourages neural models to disentangle and recombine different facets of knowledge to generalise more systematically to ne… (voir plus)w tasks. In this work, we assume that each task is associated with a subset of latent skills from an (arbitrary size) inventory. In turn, each skill corresponds to a parameter-efficient (sparse / low-rank) model adapter. By jointly learning adapters and a routing function that allocates skills to each task, the full network is instantiated as the average of the parameters of active skills. We propose several inductive biases that encourage re-usage and composition of the skills, including variable-size skill allocation and a dual-speed learning rate. We evaluate our latent-skill model in two main settings: 1) multitask reinforcement learning for instruction following on 8 levels of the BabyAI platform; and 2) few-shot fine-tuning of language models on 160 NLP tasks of the CrossFit benchmark. We find that the modular design of our network enhances sample efficiency in reinforcement learning and few-shot generalisation in supervised learning, compared to a series of baselines. These include models where parameters are fully shared, task-specific, conditionally generated (HyperFormer), or sparse mixture-of-experts (TaskMoE).
2022-12-31
Conference of the European Chapter of the Association for Computational Linguistics (publié)
doi.org
Conditional Flow Matching: Simulation-Free Dynamic Optimal Transport
Alexander Tong
Nikolay Malkin
Guillaume Huguet
Yanlei Zhang
Jarrid Rector-Brooks
Kilian Fatras
Guy Wolf
Yoshua Bengio
2022-12-31
arXiv.org (prépublication)
doi.org
Constant Memory Attentive Neural Processes
Leo Feng
Frederick Tung
Hossein Hajimirsadeghi
Yoshua Bengio
Mohamed Osama Ahmed
2022-12-31
arXiv.org (prépublication)
doi.org
Contrast-agnostic deep learning-based registration pipeline: Validation in spinal cord multimodal MRI data
E. Béal
J. Cohen-Adad
Medical image registration can be challenging, in that optimal solutions depend on the application domain (unimodal, multimodal, intra-subje… (voir plus)ct and inter-subject), anatomical sites (e.g., brain, spinal cord (SC) and lungs), dimensionality of the data (2D, 3D and 4D), deformation constraints (rigid, affine and nonlinear) and computational time. Solutions that could accommodate a large variety of applications while producing satisfactory results are needed. SynthMorph was recently introduced as an unsupervised deep learning–based registration method. A particularly interesting feature is that training is performed on synthetic data so that registration becomes agnostic to image contrast and anatomy. However, SynthMorph is particularly sensitive to the initial closeness of the images. In this work, we extend the SynthMorph method by developing a cascaded pipeline of two models that can accommodate large and fine deformations, respectively. We also validate this pipeline for the registration of intra-subject multimodal and inter-subject uni/multimodal MRI data of the SC. This task is known to be particularly difficult due to the vicinity of multiple tissue types whose morphometrics can vary substantially across subjects and contrasts. Evaluation of the method was conducted on a publicly available dataset (spine-generic, 267 subjects) and was compared with a state-of-the-art benchmark: Spinal Cord Toolbox and Advanced Normalization Tools. Results demonstrate better registration accuracy compared with the benchmark and about 24–30 times faster on CPUs depending on the image size. This proposed pipeline provides an easy-to-use, accurate and fast solution for multimodal 3D registration. The code and trained models are freely available at https://github.com/ivadomed/multimodal-registration.
2022-12-31
Aperture Neuro (publié)
doi.org
Contrastive Positive Unlabeled Learning
Anish Acharya
Sujay Sanghavi
Li Jing
Bhargav Bhushanam
Michael G. Rabbat
I. Dhillon
Self-supervised pretraining on unlabeled data followed by supervised fine-tuning on labeled data is a popular paradigm for learning from lim… (voir plus)ited labeled examples. We extend this paradigm to the classical positive unlabeled (PU) setting, where the task is to learn a binary classifier given only a few labeled positive samples, and (often) a large amount of unlabeled samples (which could be positive or negative). We first propose a simple extension of standard infoNCE family of contrastive losses, to the PU setting; and show that this learns superior representations, as compared to existing unsupervised and supervised approaches. We then develop a simple methodology to pseudo-label the unlabeled samples using a new PU-specific clustering scheme; these pseudo-labels can then be used to train the final (positive vs. negative) classifier. Our method handily outperforms state-of-the-art PU methods over several standard PU benchmark datasets, while not requiring a-priori knowledge of any class prior (which is a common assumption in other PU methods). We also provide a simple theoretical analysis that motivates our methods.
2022-12-31
(publié)
www.semanticscholar.org
Cutting Planes from the Branch-and-Bound Tree: Challenges and Opportunities
Claudio Contardo
Andrea Lodi
Andrea Tramontani
2022-12-31
INFORMS J. Comput. (publié)
doi.org
Deep Networks as Paths on the Manifold of Neural Representations
Richard D Lange
Devin Kwok
Jordan Kyle Matelsky
Xinyue Wang
David Rolnick
Konrad Paul Kording
2022-12-31
TAG-ML (publié)
proceedings.mlr.press
proceedings.mlr.press
DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization
Xinlin Li
Bang Liu
Rui Heng Yang
Vanessa Courville
Chao Xing
Vahid Partovi Nia
Efficiently deploying deep neural networks on low-resource edge devices is challenging due to their ever-increasing resource requirements. T… (voir plus)o address this issue, researchers have proposed multiplication-free neural networks, such as Power-of-Two quantization, or also known as Shift networks, which aim to reduce memory usage and simplify computation. However, existing low-bit Shift networks are not as accurate as their full-precision counterparts, typically suffering from limited weight range encoding schemes and quantization loss. In this paper, we propose the DenseShift network, which significantly improves the accuracy of Shift networks, achieving competitive performance to full-precision networks for vision and speech applications. In addition, we introduce a method to deploy an efficient DenseShift network using non-quantized floating-point activations, while obtaining 1.6X speed-up over existing methods. To achieve this, we demonstrate that zero-weight values in low-bit Shift networks do not contribute to model capacity and negatively impact inference computation. To address this issue, we propose a zero-free shifting mechanism that simplifies inference and increases model capacity. We further propose a sign-scale decomposition design to enhance training efficiency and a low-variance random initialization strategy to improve the model's transfer learning performance. Our extensive experiments on various computer vision and speech tasks demonstrate that DenseShift outperforms existing low-bit multiplication-free networks and achieves competitive performance compared to full-precision networks. Furthermore, our proposed approach exhibits strong transfer learning performance without a drop in accuracy. Our code was released on GitHub.
2022-12-31
IEEE/CVF Conference on Computer Vision and Pattern Recognition (publié)
doi.org
arxiv.org
Differential and Overlapping Effects between Exogenous and Endogenous Attention Shape Perceptual Facilitation during Visual Processing.
Mathieu Landry
Jason da Silva Castanheira
Karim Jerbi
Visuospatial attention is not a monolithic process and can be divided into different functional systems. In this framework, exogenous attent… (voir plus)ion reflects the involuntary orienting of attention resources following a salient event, whereas endogenous attention corresponds to voluntary orienting based on the goals and intentions of individuals. Previous work shows that these attention processes map onto distinct functional systems, yet evidence suggests that they are not fully independent. In the current work, we investigated the differential and overlapping effects of exogenous and endogenous attention on visual processing. We combined spatial cueing of visuospatial attention, EEG, and multivariate pattern analysis to examine where and when the effects of exogenous and endogenous attention were maximally different and maximally similar. Critically, multivariate pattern analysis provided new insights by examining whether classifiers trained to decode the cueing effect for one attention process (e.g., exogenous attention) can successfully decode the cueing effect for the other attention process (e.g., endogenous attention). These analyses uncovered differential and overlapping effects between exogenous and endogenous attention. Next, we combined principal component analyses, single-trial ERPs, and mediation analysis to determine whether these effects facilitate perception, as indexed by the behavioral spatial cueing effects of exogenous and endogenous attention. This approach revealed that three EEG components shape the cueing effects of exogenous and endogenous attention at various times after target onset. Altogether, our study provides a comprehensive account about how overlapping and differential processes of endogenous and exogenous relate to perceptual facilitation in the context of visuospatial attention.
2022-12-31
Journal of Cognitive Neuroscience (inconnu)
doi.org
On Dynamic Program Decompositions of Static Risk Measures
Jia Lin Hau
Erick Delage
Mohammad Ghavamzadeh
Marek Petrik
Optimizing static risk-averse objectives in Markov decision processes is challenging because they do not readily admit dynamic programming d… (voir plus)ecompositions. Prior work has proposed to use a dynamic decomposition of risk measures that help to formulate dynamic programs on an augmented state space. This paper shows that several existing decompositions are inherently inexact, contradicting several claims in the literature. In particular, we give examples that show that popular decompositions for CVaR and EVaR risk measures are strict overestimates of the true risk values. However, an exact decomposition is possible for VaR, and we give a simple proof that illustrates the fundamental difference between VaR and CVaR dynamic programming properties.
2022-12-31
arXiv.org (prépublication)
doi.org
Energy Transition Pathways for Deep Decarbonization of the Greater Montreal Region: An Energy Optimization Framework
Sajad Aliakbari Sani
Azadeh Maroufmashat
Frédéric Babonneau
Olivier Bahn
Erick Delage
Alain Haurie
Normand Mousseau
Kathleen Vaillancourt
More than half of the world’s population live in cities, and by 2050, it is expected that this proportion will reach almost 68%. These den… (voir plus)sely populated cities consume more than 75% of the world’s primary energy and are responsible for the emission of around 70% of anthropogenic carbon. Providing sustainable energy for the growing demand in cities requires multifaceted planning approach. In this study, we modeled the energy system of the Greater Montreal region to evaluate the impact of different environmental mitigation policies on the energy system of this region over a long-term period (2020–2050). In doing so, we have used the open-source optimization-based model called the Energy–Technology–Environment Model (ETEM). The ETEM is a long-term bottom–up energy model that provides insight into the best options for cities to procure energy, and satisfies useful demands while reducing carbon dioxide (CO2) emissions. Results show that, under a deep decarbonization scenario, the transportation, commercial, and residential sectors will contribute to emission reduction by 6.9, 1.6, and 1 million ton CO2-eq in 2050, respectively, compared with their 2020 levels. This is mainly achieved by (i) replacing fossil fuel cars with electric-based vehicles in private and public transportation sectors; (ii) replacing fossil fuel furnaces with electric heat pumps to satisfy heating demand in buildings; and (iii) improving the efficiency of buildings by isolating walls and roofs.
2022-12-31
Energies (publié)
doi.org
Explaining Graph Neural Networks Using Interpretable Local Surrogates
Farzaneh Heidari
Perouz Taslakian
Guillaume Rabusseau
2022-12-31
TAG-ML (publié)
proceedings.mlr.press
proceedings.mlr.press