Simon Lacoste-Julien

aristidebaratin@hotmail.com

Biography

Simon Lacoste-Julien is an associate professor at Mila – Quebec Artificial Intelligence Institute and in the Department of Computer Science and Operations Research (DIRO) at Université de Montréal. He is also a Canada CIFAR AI Chair and heads (part time) the SAIT AI Lab Montréal.

Lacoste-Julien‘s research interests are machine learning and applied mathematics, along with their applications to computer vision and natural language processing. He completed a BSc in mathematics, physics and computer science at McGill University, a PhD in computer science at UC Berkeley and a postdoc at the University of Cambridge.

After spending several years as a researcher at INRIA and the École normale supérieure in Paris, he returned to his home city of Montréal in 2016 to answer Yoshua Bengio’s call to help grow the Montréal AI ecosystem.

Current Students

Reza Babanezhad Harikandeh

Independent visiting researcher - Samsung SAIT

Aristide Baratin

Independent visiting researcher - Samsung SAIT

PhD - Université de Montréal

Collaborating Alumni - Université de Montréal

Principal supervisor :

Independent visiting researcher - Samsung SAIT

Collaborating Alumni - Université de Montréal

PhD - Université de Montréal

Yash Goyal

Independent visiting researcher - Samsung SAIT

yashgoyal.yg1@gmail.com

Meraj Hashemizadeh

Collaborating researcher - Université de Montréal

Alexia Jolicoeur-Martineau

Independent visiting researcher - Samsung SAIT

PhD - Université de Montréal

Kwon Kisoo

Independent visiting researcher - Seoul National University, Korea

Boris Knyazev

Independent visiting researcher - Université de Montréal

PhD - Université de Montréal

Jaewoo Lee

Independent visiting researcher - Pohang University of Science and Technology in Pohang, Korea

Michelle Liu

Collaborating researcher

Lucas Maes

PhD - Université de Montréal

Master's Research - Université de Montréal

Juan Ramirez

PhD - Université de Montréal

PhD - Université de Montréal

Theo Saulus

Collaborating researcher

Co-supervisor :

Dhanya Sridhar

Damien Scieur

Independent visiting researcher - Samsung SAIT

Motahareh Sohrabi

Collaborating Alumni - Université de Montréal

Helen Zhang

PhD - Université de Montréal

Yan Zhang

Independent visiting researcher - Samsung SAIT

Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation

Blog Posts

March 18, 2024

Sébastien Lachapelle

Divyat Mahajan

Ioannis Mitliagkas

Simon Lacoste-Julien

Read the article

Publications

Understanding Adam Requires Better Rotation Dependent Assumptions

Lucas Maes

Tianyue H. Zhang

Alexia Jolicoeur-Martineau

Ioannis Mitliagkas

Damien Scieur

Charles Guille-Escuret

Despite its widespread adoption, Adam's advantage over Stochastic Gradient Descent (SGD) lacks a comprehensive theoretical explanation. This… (see more) paper investigates Adam's sensitivity to rotations of the parameter space. We demonstrate that Adam's performance in training transformers degrades under random rotations of the parameter space, indicating a crucial sensitivity to the choice of basis. This reveals that conventional rotation-invariant assumptions are insufficient to capture Adam's advantages theoretically. To better understand the rotation-dependent properties that benefit Adam, we also identify structured rotations that preserve or even enhance its empirical performance. We then examine the rotation-dependent assumptions in the literature, evaluating their adequacy in explaining Adam's behavior across various rotation types. This work highlights the need for new, rotation-dependent theoretical frameworks to fully understand Adam's empirical success in modern machine learning tasks.

2024-10-25

ArXiv (preprint)

Accelerating Training with Neuron Interaction and Nowcasting Networks

Boris Knyazev

Abhinav Moudgil

Guillaume Lajoie

Eugene Belilovsky

Neural network training can be accelerated when a learnable update rule is used in lieu of classic adaptive optimizers (e.g. Adam). However,… (see more) learnable update rules can be costly and unstable to train and use. A simpler recently proposed approach to accelerate training is to use Adam for most of the optimization steps and periodically, only every few steps, nowcast (predict future) parameters. We improve this approach by Neuron interaction and Nowcasting (NiNo) networks. NiNo leverages neuron connectivity and graph neural networks to more accurately nowcast parameters by learning in a supervised way from a set of training trajectories over multiple tasks. We show that in some networks, such as Transformers, neuron connectivity is non-trivial. By accurately modeling neuron connectivity, we allow NiNo to accelerate Adam training by up to 50\% in vision and language tasks.

2024-09-06

ArXiv (preprint)

On PI Controllers for Updating Lagrange Multipliers in Constrained Optimization

Motahareh Sohrabi

Juan Ramirez

Tianyue H. Zhang

Jose Gallego-Posada

Constrained optimization offers a powerful framework to prescribe desired behaviors in neural network models. Typically, constrained problem… (see more)s are solved via their min-max Lagrangian formulations, which exhibit unstable oscillatory dynamics when optimized using gradient descent-ascent. The adoption of constrained optimization techniques in the machine learning community is currently limited by the lack of reliable, general-purpose update schemes for the Lagrange multipliers. This paper proposes the νPI algorithm and contributes an optimization perspective on Lagrange multiplier updates based on PI controllers, extending the work of Stooke, Achiam and Abbeel (2020). We provide theoretical and empirical insights explaining the inability of momentum methods to address the shortcomings of gradient descent-ascent, and contrast this with the empirical success of our proposed νPI controller. Moreover, we prove that νPI generalizes popular momentum methods for single-objective minimization. Our experiments demonstrate that νPI reliably stabilizes the multiplier dynamics and its hyperparameters enjoy robust and predictable behavior.

2024-07-08

Proceedings of the 41st International Conference on Machine Learning (published)

proceedings.mlr.press

Performative Prediction on Games and Mechanism Design

António Góis

Mehrnaz Mofakhami

Fernando P. Santos

Gauthier Gidel

2024-06-18

ICML.cc/2024/Workshop/Agentic_Markets (poster)

Promoting Exploration in Memory-Augmented Adam using Critical Momenta

Pranshu Malviya

Goncalo Mordido

Aristide Baratin

Reza Babanezhad Harikandeh

Jerry Huang

Razvan Pascanu

Sarath Chandar

Adaptive gradient-based optimizers, particularly Adam, have left their mark in training large-scale deep learning models. The strength of su… (see more)ch optimizers is that they exhibit fast convergence while being more robust to hyperparameter choice. However, they often generalize worse than non-adaptive methods. Recent studies have tied this performance gap to flat minima selection: adaptive methods tend to find solutions in sharper basins of the loss landscape, which in turn hurts generalization. To overcome this issue, we propose a new memory-augmented version of Adam that promotes exploration towards flatter minima by using a buffer of critical momentum terms during training. Intuitively, the use of the buffer makes the optimizer overshoot outside the basin of attraction if it is not wide enough. We empirically show that our method improves the performance of several variants of Adam on standard supervised language modelling and image classification tasks.

2024-06-09

TMLR (accepted)

Weight-Sharing Regularization

Mehran Shakerinava

Motahareh Sohrabi

Siamak Ravanbakhsh

2024-04-18

Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (published)

PopulAtion Parameter Averaging (PAPA)

Alexia Jolicoeur-Martineau

Emy Gervais

Kilian FATRAS

Yan Zhang

2024-04-05

TMLR (accepted)

On the Identifiability of Quantized Factors

Vitória Barin Pacela

Kartik Ahuja

Pascal Vincent

Disentanglement aims to recover meaningful latent ground-truth factors from the observed distribution solely, and is formalized through the… (see more) theory of identifiability. The identifiability of independent latent factors is proven to be impossible in the unsupervised i.i.d. setting under a general nonlinear map from factors to observations. In this work, however, we demonstrate that it is possible to recover quantized latent factors under a generic nonlinear diffeomorphism. We only assume that the latent factors have independent discontinuities in their density, without requiring the factors to be statistically independent. We introduce this novel form of identifiability, termed quantized factor identifiability, and provide a comprehensive proof of the recovery of the quantized factors.

2024-03-15

Proceedings of the Third Conference on Causal Learning and Reasoning (published)

proceedings.mlr.press

Balancing Act: Constraining Disparate Impact in Sparse Models

Meraj Hashemizadeh

Juan Ramirez

Rohan Sukumaran

Golnoosh Farnadi

Jose Gallego-Posada

Model pruning is a popular approach to enable the deployment of large deep learning models on edge devices with restricted computational or … (see more)storage capacities. Although sparse models achieve performance comparable to that of their dense counterparts at the level of the entire dataset, they exhibit high accuracy drops for some data sub-groups. Existing methods to mitigate this disparate impact induced by pruning (i) rely on surrogate metrics that address the problem indirectly and have limited interpretability; or (ii) scale poorly with the number of protected sub-groups in terms of computational cost. We propose a constrained optimization approach that directly addresses the disparate impact of pruning: our formulation bounds the accuracy change between the dense and sparse models, for each sub-group. This choice of constraints provides an interpretable success criterion to determine if a pruned model achieves acceptable disparity levels. Experimental results demonstrate that our technique scales reliably to problems involving large models and hundreds of protected sub-groups.

2024-01-16

ICLR.cc/2024/Conference (poster)

Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies

Sébastien Lachapelle

Pau Rodriguez

Yash Sharma

Katie Everett

Rémi LE PRIOL

Alexandre Lacoste

2024-01-10

ArXiv (preprint)

Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation

Sébastien Lachapelle

Divyat Mahajan

Ioannis Mitliagkas

We tackle the problems of latent variables identification and "out-of-support'' image generation in representation learning. We show that bo… (see more)th are possible for a class of decoders that we call additive, which are reminiscent of decoders used for object-centric representation learning (OCRL) and well suited for images that can be decomposed as a sum of object-specific images. We provide conditions under which exactly solving the reconstruction problem using an additive decoder is guaranteed to identify the blocks of latent variables up to permutation and block-wise invertible transformations. This guarantee relies only on very weak assumptions about the distribution of the latent factors, which might present statistical dependencies and have an almost arbitrarily shaped support. Our result provides a new setting where nonlinear independent component analysis (ICA) is possible and adds to our theoretical understanding of OCRL methods. We also show theoretically that additive decoders can generate novel images by recombining observed factors of variations in novel ways, an ability we refer to as Cartesian-product extrapolation. We show empirically that additivity is crucial for both identifiability and extrapolation on simulated data.

Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?

Boris Knyazev

DOHA HWANG

2023-07-03

Proceedings of the 40th International Conference on Machine Learning (published)