Publications

Histology-informed automatic parcellation of white matter tracts in the rat spinal cord
Harris Nami
Christian S. Perone
Julien Cohen-Adad
The white matter is organized into “tracts” or “bundles,” which connect different parts of the central nervous system. Knowing where… (voir plus) these tracts are located in each individual is important for understanding the cause of potential sensorial, motor or cognitive deficits and for developing appropriate treatments. Traditionally, tracts are found using tracer injection, which is a difficult, slow and poorly scalable technique. However, axon populations from a given tract exhibit specific characteristics in terms of morphometrics and myelination. Hence, the delineation of tracts could, in principle, be done based on their morphometry. The objective of this study was to generate automatic parcellation of the rat spinal white matter tracts using the manifold information from scanning electron microscopy images of the entire spinal cord. The axon morphometrics (axon density, axon diameter, myelin thickness and g-ratio) were computed pixelwise following automatic axon segmentation using AxonSeg. The parcellation was based on an agglomerative clustering algorithm to group the tracts. Results show that axon morphometrics provide sufficient information to automatically identify some white matter tracts in the spinal cord, however, not all tracts were correctly identified. Future developments of microstructure quantitative MRI even bring hope for a personalized clustering of white matter tracts in each individual patient. The generated atlas and the associated code can be found at https://github.com/neuropoly/tract-clustering.
2022-11-28
Frontiers in Neuroanatomy (publié)
doi.org
Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions
Courtney Paquette
Elliot Paquette
Ben Adlam
Jeffrey Pennington
Stochastic gradient descent (SGD) is a pillar of modern machine learning, serving as the go-to optimization algorithm for a diverse array of… (voir plus) problems. While the empirical success of SGD is often attributed to its computational efficiency and favorable generalization behavior, neither effect is well understood and disentangling them remains an open problem. Even in the simple setting of convex quadratic problems, worst-case analyses give an asymptotic convergence rate for SGD that is no better than full-batch gradient descent (GD), and the purported implicit regularization effects of SGD lack a precise explanation. In this work, we study the dynamics of multi-pass SGD on high-dimensional convex quadratics and establish an asymptotic equivalence to a stochastic differential equation, which we call homogenized stochastic gradient descent (HSGD), whose solutions we characterize explicitly in terms of a Volterra integral equation. These results yield precise formulas for the learning and risk trajectories, which reveal a mechanism of implicit conditioning that explains the efficiency of SGD relative to GD. We also prove that the noise from SGD negatively impacts generalization performance, ruling out the possibility of any type of implicit regularization in this context. Finally, we show how to adapt the HSGD formalism to include streaming SGD, which allows us to produce an exact prediction for the excess risk of multi-pass SGD relative to that of streaming SGD (bootstrap risk).
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net
Improving the accuracy of single-trial fMRI response estimates using GLMsingle
Jacob S Prince
Ian Charest
Jan W Kurzawski
John A Pyles
Michael J Tarr
Kendrick Kay
2022-11-28
eLife (publié)
doi.org
Inductive Logical Query Answering in Knowledge Graphs
Mikhail Galkin
Zhaocheng Zhu
Hongyu Ren
Jian Tang
Formulating and answering logical queries is a standard communication interface for knowledge graphs (KGs). Alleviating the notorious incomp… (voir plus)leteness of real-world KGs, neural methods achieved impressive results in link prediction and complex query answering tasks by learning representations of entities, relations, and queries. Still, most existing query answering methods rely on transductive entity embeddings and cannot generalize to KGs containing new entities without retraining the entity embeddings. In this work, we study the inductive query answering task where inference is performed on a graph containing new entities with queries over both seen and unseen entities. To this end, we devise two mechanisms leveraging inductive node and relational structure representations powered by graph neural networks (GNNs). Experimentally, we show that inductive models are able to perform logical reasoning at inference time over unseen nodes generalizing to graphs up to 500% larger than training ones. Exploring the efficiency--effectiveness trade-off, we find the inductive relational structure representation method generally achieves higher performance, while the inductive node representation method is able to answer complex queries in the inference-only regime without any training on queries and scales to graphs of millions of nodes. Code is available at https://github.com/DeepGraphLearning/InductiveQE.
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net
Learning Robust Dynamics through Variational Sparse Gating
Arnav Kumar Jain
Shivakanth Sujit
Shruti Joshi
Vincent Michalski
Danijar Hafner
Samira Ebrahimi-Kahou
Learning world models from their sensory inputs enables agents to plan for actions by imagining their future outcomes. World models have pre… (voir plus)viously been shown to improve sample-efficiency in simulated environments with few objects, but have not yet been applied successfully to environments with many objects. In environments with many objects, often only a small number of them are moving or interacting at the same time. In this paper, we investigate integrating this inductive bias of sparse interactions into the latent dynamics of world models trained from pixels. First, we introduce Variational Sparse Gating (VSG), a latent dynamics model that updates its feature dimensions sparsely through stochastic binary gates. Moreover, we propose a simplified architecture Simple Variational Sparse Gating (SVSG) that removes the deterministic pathway of previous models, resulting in a fully stochastic transition function that leverages the VSG mechanism. We evaluate the two model architectures in the BringBackShapes (BBS) environment that features a large number of moving objects and partial observability, demonstrating clear improvements over prior models.
2022-11-28
Advances in Neural Information Processing Systems (Accept)
doi.org
openreview.net
Long Range Graph Benchmark
Vijay Prakash Dwivedi
Ladislav Rampasek
Mikhail Galkin
Ali Parviz
Guy Wolf
Anh Tuan Luu
Dominique Beaini
Graph Neural Networks (GNNs) that are based on the message passing (MP) paradigm generally exchange information between 1-hop neighbors to b… (voir plus)uild node representations at each layer. In principle, such networks are not able to capture long-range interactions (LRI) that may be desired or necessary for learning a given task on graphs. Recently, there has been an increasing interest in development of Transformer-based methods for graphs that can consider full node connectivity beyond the original sparse structure, thus enabling the modeling of LRI. However, MP-GNNs that simply rely on 1-hop message passing often fare better in several existing graph benchmarks when combined with positional feature representations, among other innovations, hence limiting the perceived utility and ranking of Transformer-like architectures. Here, we present the Long Range Graph Benchmark (LRGB) with 5 graph learning datasets: PascalVOC-SP, COCO-SP, PCQM-Contact, Peptides-func and Peptides-struct that arguably require LRI reasoning to achieve strong performance in a given task. We benchmark both baseline GNNs and Graph Transformer networks to verify that the models which capture long-range dependencies perform significantly better on these tasks. Therefore, these datasets are suitable for benchmarking and exploration of MP-GNNs and Graph Transformer architectures that are intended to capture LRI.
2022-11-28
Neural Information Processing Systems (Accept)
doi.org
openreview.net
MAgNet: Mesh Agnostic Neural PDE Solver
Oussama Boussif
Dan Assouline
Loubna Benabbou
Yoshua Bengio
The computational complexity of classical numerical methods for solving Partial Differential Equations (PDE) scales significantly as the res… (voir plus)olution increases. As an important example, climate predictions require fine spatio-temporal resolutions to resolve all turbulent scales in the fluid simulations. This makes the task of accurately resolving these scales computationally out of reach even with modern supercomputers. As a result, current numerical modelers solve PDEs on grids that are too coarse (3km to 200km on each side), which hinders the accuracy and usefulness of the predictions. In this paper, we leverage the recent advances in Implicit Neural Representations (INR) to design a novel architecture that predicts the spatially continuous solution of a PDE given a spatial position query. By augmenting coordinate-based architectures with Graph Neural Networks (GNN), we enable zero-shot generalization to new non-uniform meshes and long-term predictions up to 250 frames ahead that are physically consistent. Our Mesh Agnostic Neural PDE Solver (MAgNet) is able to make accurate predictions across a variety of PDE simulation datasets and compares favorably with existing baselines. Moreover, MAgNet generalizes well to different meshes and resolutions up to four times those trained on.
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation
Vikram Voleti
Alexia Jolicoeur-Martineau
Christopher Pal
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net
Is a Modular Architecture Enough?
Sarthak Mittal
Yoshua Bengio
Guillaume Lajoie
Inspired from human cognition, machine learning systems are gradually revealing advantages of sparser and more modular architectures. Recent… (voir plus) work demonstrates that not only do some modular architectures generalize well, but they also lead to better out-of-distribution generalization, scaling properties, learning speed, and interpretability. A key intuition behind the success of such systems is that the data generating system for most real-world settings is considered to consist of sparsely interacting parts, and endowing models with similar inductive biases will be helpful. However, the field has been lacking in a rigorous quantitative assessment of such systems because these real-world data distributions are complex and unknown. In this work, we provide a thorough assessment of common modular architectures, through the lens of simple and known modular data distributions. We highlight the benefits of modularity and sparsity and reveal insights on the challenges faced while optimizing modular systems. In doing so, we propose evaluation metrics that highlight the benefits of modularity, the regimes in which these benefits are substantial, as well as the sub-optimality of current end-to-end learned modular systems as opposed to their claimed potential.
2022-11-28
Neural Information Processing Systems (Accept)
doi.org
openreview.net
Myriad: a real-world testbed to bridge trajectory optimization and deep learning
Nikolaus H. R. Howe
Simon Dufort-Labbé
Nitarshan Rajkumar
Pierre-Luc Bacon
We present Myriad, a testbed written in JAX for learning and planning in real-world continuous environments. The primary contributions of My… (voir plus)riad are threefold. First, Myriad provides machine learning practitioners access to trajectory optimization techniques for application within a typical automatic differentiation workflow. Second, Myriad presents many real-world optimal control problems, ranging from biology to medicine to engineering, for use by the machine learning community. Formulated in continuous space and time, these environments retain some of the complexity of real-world systems often abstracted away by standard benchmarks. As such, Myriad strives to serve as a stepping stone towards application of modern machine learning techniques for impactful real-world tasks. Finally, we use the Myriad repository to showcase a novel approach for learning and control tasks. Trained in a fully end-to-end fashion, our model leverages an implicit planning module over neural ordinary differential equations, enabling simultaneous learning and planning with complex environment dynamics.
2022-11-28
Neural Information Processing Systems (Accept)
doi.org
openreview.net
Neural Attentive Circuits
Nasim Rahaman
Martin Weiss
Francesco Locatello
Chris Pal
Yoshua Bengio
Bernhard Schölkopf
Li Erran Li
Nicolas Ballas
Recent work has seen the development of general purpose neural architectures that can be trained to perform tasks across diverse data modali… (voir plus)ties. General purpose models typically make few assumptions about the underlying data-structure and are known to perform well in the large-data regime. At the same time, there has been growing interest in modular neural architectures that represent the data using sparsely interacting modules. These models can be more robust out-of-distribution, computationally efficient, and capable of sample-efficient adaptation to new data. However, they tend to make domain-specific assumptions about the data, and present challenges in how module behavior (i.e., parameterization) and connectivity (i.e., their layout) can be jointly learned. In this work, we introduce a general purpose, yet modular neural architecture called Neural Attentive Circuits (NACs) that jointly learns the parameterization and a sparse connectivity of neural modules without using domain knowledge. NACs are best understood as the combination of two systems that are jointly trained end-to-end: one that determines the module configuration and the other that executes it on an input. We demonstrate qualitatively that NACs learn diverse and meaningful module configurations on the NLVR2 dataset without additional supervision. Quantitatively, we show that by incorporating modularity in this way, NACs improve upon a strong non-modular baseline in terms of low-shot adaptation on CIFAR and CUBs dataset by about 10%, and OOD robustness on Tiny ImageNet-R by about 2.5%. Further, we find that NACs can achieve an 8x speedup at inference time while losing less than 3% performance. Finally, we find NACs to yield competitive results on diverse data modalities spanning point-cloud classification, symbolic processing and text-classification from ASCII bytes, thereby confirming its general purpose nature.
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net
PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding
Minghao Xu
Zuobai Zhang
Jiarui Lu
Zhaocheng Zhu
Yangtian Zhang
Chang Ma
Runcheng Liu
Jian Tang
We are now witnessing significant progress of deep learning methods in a variety of tasks (or datasets) of proteins. However, there is a lac… (voir plus)k of a standard benchmark to evaluate the performance of different methods, which hinders the progress of deep learning in this field. In this paper, we propose such a benchmark called PEER, a comprehensive and multi-task benchmark for Protein sEquence undERstanding. PEER provides a set of diverse protein understanding tasks including protein function prediction, protein localization prediction, protein structure prediction, protein-protein interaction prediction, and protein-ligand interaction prediction. We evaluate different types of sequence-based methods for each task including traditional feature engineering approaches, different sequence encoding methods as well as large-scale pre-trained protein language models. In addition, we also investigate the performance of these methods under the multi-task learning setting. Experimental results show that large-scale pre-trained protein language models achieve the best performance for most individual tasks, and jointly training multiple tasks further boosts the performance. The datasets and source codes of this benchmark are all available at https://github.com/DeepGraphLearning/PEER_Benchmark
2022-11-28
Conference on Neural Information Processing Systems (Accept)
doi.org
openreview.net