Guillaume Lajoie

matteo.alleman@mila.quebec

Biographie

Guillaume Lajoie est professeur agrégé au Département de mathématiques et de statistiques (DMS) de l'Université de Montréal et membre académique principal de Mila – Institut québécois d’intelligence artificielle. Il est titulaire d'une chaire CIFAR (CCAI Canada) ainsi que d'une chaire de recherche du Canada (CRC) en calcul et interfaçage neuronaux.

Ses recherches sont positionnées à l'intersection de l'IA et des neurosciences où il développe des outils pour mieux comprendre les mécanismes d'intelligence communs aux systèmes biologiques et artificiels. Les contributions de son groupe de recherche vont des progrès des paradigmes d'apprentissage à plusieurs échelles pour les grands systèmes artificiels aux applications en neurotechnologie. Dr. Lajoie participe activement aux efforts de développement responsables de l'IA, cherchant à identifier les lignes directrices et les meilleures pratiques pour l'utilisation de l'IA dans la recherche et au-delà.

Étudiants actuels

Matteo Alleman

Postdoctorat - UdeM

Rohan Banerjee

Collaborateur·rice alumni - Polytechnique

rohan.banerjee@mila.quebec

Visiteur de recherche indépendant - N/A

philippe.beaudoin@mila.quebec

Doctorat - UdeM

Co-superviseur⋅e :

Hugo Larochelle

sangnie.bhardwaj@mila.quebec

Colin Bredenberg

Visiteur de recherche indépendant - UdeM

Co-superviseur⋅e :

colin.bredenberg@mila.quebec

Leo Choiniere

Doctorat - UdeM

leo.choiniere@mila.quebec

Claudia Clopath

Visiteur de recherche indépendant - University College London

claudia.clopath@mila.quebec

Eric Elmoznino

Doctorat - UdeM

Superviseur⋅e principal⋅e :

Yoshua Bengio

eric.elmoznino@mila.quebec

Leo Gagnon

Doctorat - UdeM

leo.gagnon@mila.quebec

Tom George

Postdoctorat - McGill

Superviseur⋅e principal⋅e :

tomgeorge1@btinternet.com

Juan Guerra

Doctorat - Polytechnique

juan.guerra@mila.quebec

thomas.jiralerspong@mila.quebec

Nanda Harishankar Krishna

Doctorat - UdeM

Anna Jahn

Visiteur de recherche indépendant - McGill

anna.jahn1@mila.quebec

Chen Jiang

Doctorat - McGill

Superviseur⋅e principal⋅e :

Paul Masset

chen.jiang@mila.quebec

Thomas Jiralerspong

Doctorat - UdeM

Co-superviseur⋅e :

Yoshua Bengio

Maîtrise recherche - UdeM

Co-superviseur⋅e :

Dhanya Sridhar

tejaskasetty@gmail.com

Doctorat - McGill

Superviseur⋅e principal⋅e :

pingsheng.li@mila.quebec

mathys.loiselle@mila.quebec

Mathys Loiselle

Stagiaire de recherche - Concordia

Co-superviseur⋅e :

Matt Perich

Ximeng Mao

Doctorat - UdeM

Co-superviseur⋅e :

Joelle Pineau

ximeng.mao@mila.quebec

Abdel Mfougouon Njupoun

Doctorat - UdeM

Co-superviseur⋅e :

alexandre.payeur@mila.quebec

abdelnjupoun@gmail.com

Doctorat - UdeM

Co-superviseur⋅e :

Visiteur de recherche indépendant - UdeM

Julia Price

Maîtrise recherche - UdeM

julia.price@mila.quebec

Mauricio Rivera

Maîtrise recherche - UdeM

Superviseur⋅e principal⋅e :

Marco Bonizzato

mauriv2010@gmail.com

hee-woon.ryoo@mila.quebec

Avery Ryoo

Doctorat - UdeM

Superviseur⋅e principal⋅e :

Matt Perich

Doctorat - UdeM

Co-superviseur⋅e :

Lune Bellec

sainathp@mila.quebec

ezekiel.williams@mila.quebec

Ryan Vogt

Postdoctorat - UdeM

ryan.vogt@mila.quebec

Ezekiel Williams

Doctorat - UdeM

Visiteur de recherche indépendant - University of Western Ontario

jinkang.xiang@mila.quebec

Billets de blogue

Représentation graphique d'un nerf vague

21 mai 2025

Apprentissage automatique pour la segmentation des différentes activations des fibres nerveuses à partir des signaux neuronaux du cerveau vers le corps

par

Param Raval

Olivier Tessier-Larivière

Pascal Fortier-Poisson

Blake Richards

Guillaume Lajoie

Lire l'article

13 juin 2024

Que nous apprennent les distributions des coefficients synaptiques au sujet de l’apprentissage dans le cerveau ?

par

Roman Pogodin

Jonathan Cornford

Arna Ghosh

Gauthier Gidel

Guillaume Lajoie

Blake Richards

Lire l'article

Publications

Accelerating Training with Neuron Interaction and Nowcasting Networks

Neural network training can be accelerated when a learnable update rule is used in lieu of classic adaptive optimizers (e.g. Adam). However,… (voir plus) learnable update rules can be costly and unstable to train and use. Recently, Jang et al. (2023) proposed a simpler approach to accelerate training based on weight nowcaster networks (WNNs). In their approach, Adam is used for most of the optimization steps and periodically, only every few steps, a WNN nowcasts (predicts near future) parameters. We improve WNNs by proposing neuron interaction and nowcasting (NiNo) networks. In contrast to WNNs, NiNo leverages neuron connectivity and graph neural networks to more accurately nowcast parameters. We further show that in some networks, such as Transformers, modeling neuron connectivity accurately is challenging. We address this and other limitations, which allows NiNo to accelerate Adam training by up to 50% in vision and language tasks.

2025-04-22

International Conference on Learning Representations (Accept (Poster))

Expressivity of Neural Networks with Random Weights and Learned Biases

Ezekiel Williams

Avery Hee-Woon Ryoo

Thomas Jiralerspong

Alexandre Payeur

Matthew G Perich

Luca Mazzucato

Landmark universal function approximation results for neural networks with trained weights and biases provided the impetus for the ubiquitou… (voir plus)s use of neural networks as learning models in neuroscience and Artificial Intelligence (AI). Recent work has extended these results to networks in which a smaller subset of weights (e.g., output weights) are tuned, leaving other parameters random. However, it remains an open question whether universal approximation holds when only biases are learned, despite evidence from neuroscience and AI that biases significantly shape neural responses. The current paper answers this question. We provide theoretical and numerical evidence demonstrating that feedforward neural networks with fixed random weights can approximate any continuous function on compact sets. We further show an analogous result for the approximation of dynamical systems with recurrent neural networks. Our findings are relevant to neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights, as well as for AI, where they shed light on recent fine-tuning methods for large language models, like bias and prefix-based approaches.

2025-04-22

International Conference on Learning Representations (Accept (Poster))

Multi-Agent Cooperation Through Learning-Aware Policy Gradients

Alexander Meulemans

Seijin Kobayashi

Johannes Von Oswald

Nino Scherrer

Eric Elmoznino

Blake A. Richards

Blaise Agüera y Arcas

João Sacramento

Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning. How can we achieve cooperation… (voir plus) among self-interested, independent learning agents? Promising recent work has shown that in certain tasks cooperation can be established between learning-aware agents who model the learning dynamics of each other. Here, we present the first unbiased, higher-derivative-free policy gradient algorithm for learning-aware reinforcement learning, which takes into account that other agents are themselves learning through trial and error based on multiple noisy trials. We then leverage efficient sequence models to condition behavior on long observation histories that contain traces of the learning dynamics of other agents. Training long-context policies with our algorithm leads to cooperative behavior and high returns on standard social dilemmas, including a challenging environment where temporally-extended action coordination is required. Finally, we derive from the iterated prisoner's dilemma a novel explanation for how and when cooperation arises among self-interested learning-aware agents.

2025-04-22

International Conference on Learning Representations (Accept (Poster))

Latent Representation Learning for Multimodal Brain Activity Translation

Arman Afrasiyabi

Dhananjay Bhaskar

Erica Lindsey Busch

Laurent Caplette

Rahul Singh

Nicholas B Turk-Browne

Smita Krishnaswamy

Neuroscience employs diverse neuroimaging techniques, each offering distinct insights into brain activity, from electrophysiological recordi… (voir plus)ngs such as EEG, which have high temporal resolution, to hemodynamic modalities such as fMRI, which have increased spatial precision. However, integrating these heterogeneous data sources remains a challenge, which limits a comprehensive understanding of brain function. We present the Spatiotemporal Alignment of Multimodal Brain Activity (SAMBA) framework, which bridges the spatial and temporal resolution gaps across modalities by learning a unified latent space free of modality-specific biases. SAMBA introduces a novel attention-based wavelet decomposition for spectral filtering of electrophysiological recordings, graph attention networks to model functional connectivity between functional brain units, and recurrent layers to capture temporal autocorrelations in brain signal. We show that the training of SAMBA, aside from achieving translation, also learns a rich representation of brain information processing. We showcase this classify external stimuli driving brain activity from the representation learned in hidden layers of SAMBA, paving the way for broad downstream applications in neuroscience research and clinical contexts.

2025-04-05

ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (publié)

Accelerated learning of a noninvasive human brain-computer interface via manifold geometry

Erica Lindsey Busch

E. Chandra Fincke

Smita Krishnaswamy

Nicholas B Turk-Browne

2025-04-02

bioRxiv (prépublication)

In-Context Parametric Inference: Point or Distribution Estimators?

Sarthak Mittal

Yoshua Bengio

Nikolay Malkin

Bayesian and frequentist inference are two fundamental paradigms in statistical estimation. Bayesian methods treat hypotheses as random vari… (voir plus)ables, incorporating priors and updating beliefs via Bayes' theorem, whereas frequentist methods assume fixed but unknown hypotheses, relying on estimators like maximum likelihood. While extensive research has compared these approaches, the frequentist paradigm of obtaining point estimates has become predominant in deep learning, as Bayesian inference is challenging due to the computational complexity and the approximation gap of posterior estimation methods. However, a good understanding of trade-offs between the two approaches is lacking in the regime of amortized estimators, where in-context learners are trained to estimate either point values via maximum likelihood or maximum a posteriori estimation, or full posteriors using normalizing flows, score-based diffusion samplers, or diagonal Gaussian approximations, conditioned on observations. To help resolve this, we conduct a rigorous comparative analysis spanning diverse problem settings, from linear models to shallow neural networks, with a robust evaluation framework assessing both in-distribution and out-of-distribution generalization on tractable tasks. Our experiments indicate that amortized point estimators generally outperform posterior inference, though the latter remain competitive in some low-dimensional problems, and we further discuss why this might be the case.

2025-02-16

ArXiv (prépublication)

Amortized In-Context Bayesian Posterior Estimation

Sarthak Mittal

N. L. Bracher

Priyank Jaini

Marcus Brubaker

Bayesian inference provides a natural way of incorporating prior beliefs and assigning a probability measure to the space of hypotheses. Cur… (voir plus)rent solutions rely on iterative routines like Markov Chain Monte Carlo (MCMC) sampling and Variational Inference (VI), which need to be re-run whenever new observations are available. Amortization, through conditional estimation, is a viable strategy to alleviate such difficulties and has been the guiding principle behind simulation-based inference, neural processes and in-context methods using pre-trained models. In this work, we conduct a thorough comparative analysis of amortized in-context Bayesian posterior estimation methods from the lens of different optimization objectives and architectural choices. Such methods train an amortized estimator to perform posterior parameter inference by conditioning on a set of data examples passed as context to a sequence model such as a transformer. In contrast to language models, we leverage permutation invariant architectures as the true posterior is invariant to the ordering of context examples. Our empirical study includes generalization to out-of-distribution tasks, cases where the assumed underlying model is misspecified, and transfer from simulated to real problems. Subsequently, it highlights the superiority of the reverse KL estimator for predictive problems, especially when combined with the transformer architecture and normalizing flows.

2025-02-09

ArXiv (prépublication)

Robust prior-biased acquisition function for human-in-the-loop Bayesian optimization.

Rose Guay-Hottin

Lison Kardassevitch

Hugo Pham

Marco Bonizzato

2025-01-22

Knowledge-Based Systems (publié)

Neural networks with optimized single-neuron adaptation uncover biologically plausible regularization

Neurons in the brain have rich and adaptive input-output properties. Features such as heterogeneous f-I curves and spike frequency adaptatio… (voir plus)n are known to place single neurons in optimal coding regimes when facing changing stimuli. Yet, it is still unclear how brain circuits exploit single-neuron flexibility, and how network-level requirements may have shaped such cellular function. To answer this question, a multi-scaled approach is needed where the computations of single neurons and neural circuits must be considered as a complete system. In this work, we use artificial neural networks to systematically investigate single-neuron input-output adaptive mechanisms, optimized in an end-to-end fashion. Throughout the optimization process, each neuron has the liberty to modify its nonlinear activation function, parametrized to mimic f-I curves of biological neurons, and to learn adaptation strategies to modify activation functions in real-time during a task. We find that such networks show much-improved robustness to noise and changes in input statistics. Importantly, we find that this procedure recovers precise coding strategies found in biological neurons, such as gain scaling and fractional order differentiation/integration. Using tools from dynamical systems theory, we analyze the role of these emergent single-neuron properties and argue that neural diversity and adaptation play an active regularization role, enabling neural circuits to optimally propagate information across time.

2024-12-12

PLOS Computational Biology (publié)

Brain-like learning with exponentiated gradients

Jonathan Cornford

Roman Pogodin

Arna Ghosh

Kaiwen Sheng

Brendan A. Bicknell

Olivier Codol

Beverley A. Clark

Blake A. Richards

Computational neuroscience relies on gradient descent (GD) for training artificial neural network (ANN) models of the brain. The advantage o… (voir plus)f GD is that it is effective at learning difficult tasks. However, it produces ANNs that are a poor phenomenological fit to biology, making them less relevant as models of the brain. Specifically, it violates Dale’s law, by allowing synapses to change from excitatory to inhibitory, and leads to synaptic weights that are not log-normally distributed, contradicting experimental data. Here, starting from first principles of optimisation theory, we present an alternative learning algorithm, exponentiated gradient (EG), that respects Dale’s Law and produces log-normal weights, without losing the power of learning with gradients. We also show that in biologically relevant settings EG outperforms GD, including learning from sparsely relevant signals and dealing with synaptic pruning. Altogether, our results show that EG is a superior learning algorithm for modelling the brain with ANNs.

2024-10-25

bioRxiv (prépublication)

A Complexity-Based Theory of Compositionality

2024-10-17

ArXiv (prépublication)

Learning Stochastic Rainbow Networks

Vivian White

Muawiz Sajjad Chaudhary

Guy Wolf

Kameron Decker Harris

Random feature models are a popular approach for studying network learning that can capture important behaviors while remaining simpler than… (voir plus) traditional training. Guth et al. [2024] introduced “rainbow” networks which model the distribution of trained weights as correlated random features conditioned on previous layer activity. Sampling new weights from distributions fit to learned networks led to similar performance in entirely untrained networks, and the observed weight covariance were found to be low rank. This provided evidence that random feature models could be extended to some networks away from initialization, but White et al. [2024] failed to replicate their results in the deeper ResNet18 architecture. Here we ask whether the rainbow formulation can succeed in deeper networks by directly training a stochastic ensemble of random features, which we call stochastic rainbow networks. At every gradient descent iteration, new weights are sampled for all intermediate layers and features aligned layer-wise. We find: (1) this approach scales to deeper models, which outperform shallow networks at large widths; (2) ensembling multiple samples from the stochastic model is better than retraining the classifier head; and (3) low-rank parameterization of the learnable weight covariances can approach the accuracy of full-rank networks. This offers more evidence for rainbow and other structured random feature networks as reduced models of deep learning.

2024-10-09

NeurIPS.cc/2024/Workshop/SciForDL (poster)