Simon Lacoste-Julien

aristidebaratin@hotmail.com

Biography

Simon Lacoste-Julien is an associate professor at Mila – Quebec Artificial Intelligence Institute and in the Department of Computer Science and Operations Research (DIRO) at Université de Montréal. He is also a Canada CIFAR AI Chair and heads (part time) the SAIT AI Lab Montréal.

Lacoste-Julien‘s research interests are machine learning and applied mathematics, along with their applications to computer vision and natural language processing. He completed a BSc in mathematics, physics and computer science at McGill University, a PhD in computer science at UC Berkeley and a postdoc at the University of Cambridge.

After spending several years as a researcher at INRIA and the École normale supérieure in Paris, he returned to his home city of Montréal in 2016 to answer Yoshua Bengio’s call to help grow the Montréal AI ecosystem.

Current Students

Reza Babanezhad Harikandeh

Independent visiting researcher - Samsung SAIT

babanezr@mila.quebec

Aristide Baratin

Independent visiting researcher - Samsung SAIT

PhD - Université de Montréal

vitoria.barin-pacela@mila.quebec

PhD - Université de Montréal

simon.dufort-labbe@mila.quebec

Marwa El Halabi

Independent visiting researcher - Samsung SAIT

marwa.el-halabi@mila.quebec

PhD - Université de Montréal

antonio-miguel.gois@mila.quebec

Yash Goyal

Independent visiting researcher - Samsung SAIT

yashgoyal.yg1@gmail.com

Meraj Hashemizadeh

Collaborating researcher - Université de Montréal

merajhse@mila.quebec

fahimeh.hosseini@mila.quebec

Fahimeh HosseiniNoohdani

Collaborating researcher - Université de Montréal

pedram.khorsandi@mila.quebec

Pedram Khorsandi

PhD - Université de Montréal

Boris Knyazev

Independent visiting researcher - Université de Montréal

Independent visiting researcher - Samsung - SAIT

lachaseb@mila.quebec

yukti.makhija@mila.quebec

Lucas Maes

PhD - Université de Montréal

lucas.maes@mila.quebec

PhD - Université de Montréal

Co-supervisor :

Ioannis Mitliagkas

PhD - Université de Montréal

Co-supervisor :

Aaron Courville

alan.milligan@mila.quebec

Collaborating Alumni - Université de Montréal

rozhin.nobahari@mila.quebec

Juan Ramirez

PhD - Université de Montréal

juan.ramirez@mila.quebec

PhD - Université de Montréal

mansi.rankawat@mila.quebec

Independent visiting researcher - Univeristy of Tübingen

Theo Saulus

PhD - Université de Montréal

Co-supervisor :

Dhanya Sridhar

theo.saulus@mila.quebec

motahareh.sohrabi@mila.quebec

Damien Scieur

Independent visiting researcher - Samsung SAIT

Collaborating researcher - Université de Montréal

tianyue.zhang@mila.quebec

Tianyue (Helen) Zhang

PhD - Université de Montréal

Yan Zhang

Independent visiting researcher - Samsung SAIT

yan.zhang@mila.quebec

Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation

Blog Posts

March 18, 2024

Sébastien Lachapelle

Divyat Mahajan

Ioannis Mitliagkas

Simon Lacoste-Julien

Read the article

Publications

Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation

Si Yi Meng

Sharan Vaswani

Issam Hadj Laradji

Mark Schmidt

We consider stochastic second-order methods for minimizing smooth and strongly-convex functions under an interpolation condition satisfied b… (see more)y over-parameterized models. Under this condition, we show that the regularized subsampled Newton method (R-SSN) achieves global linear convergence with an adaptive step-size and a constant batch-size. By growing the batch size for both the subsampled gradient and Hessian, we show that R-SSN can converge at a quadratic rate in a local neighbourhood of the solution. We also show that R-SSN attains local linear convergence for the family of self-concordant functions. Furthermore, we analyze stochastic BFGS algorithms in the interpolation setting and prove their global linear convergence. We empirically evaluate stochastic L-BFGS and a "Hessian-free" implementation of R-SSN for binary classification on synthetic, linearly-separable datasets and real datasets under a kernel mapping. Our experimental results demonstrate the fast convergence of these methods, both in terms of the number of iterations and wall-clock time.

2019-12-31

AISTATS (published)

GAIT: A Geometric Approach to Information Theory

Jose Gallego-Posada

Ankit Vani

Max Schwarzer

We advocate the use of a notion of entropy that reflects the relative abundances of the symbols in an alphabet, as well as the similarities … (see more)between them. This concept was originally introduced in theoretical ecology to study the diversity of ecosystems. Based on this notion of entropy, we introduce geometry-aware counterparts for several concepts and theorems in information theory. Notably, our proposed divergence exhibits performance on par with state-of-the-art methods based on the Wasserstein distance, but enjoys a closed-form expression that can be computed efficiently. We demonstrate the versatility of our method via experiments on a broad range of domains: training generative models, computing image barycenters, approximating empirical measures and counting modes.

2019-12-31

AISTATS (published)

How to make your optimizer generalize better

Sharan Vaswani

Reza Babenzhad

Sait AI Lab

Montreal

Jose Gallego

Aaron Mishkin

Nicolas Roux

We study the implicit regularization of optimization methods for linear models interpolating the training data in the under-parametrized and… (see more) over-parametrized regimes. For over-parameterized linear regression, where there are inﬁnitely many interpolating solutions, different optimization methods can converge to solutions with varying generalization performance. In this setting, we show that projections onto linear spans can be used to move between solutions. Furthermore, via a simple reparameterization, we can ensure that an arbitrary optimizer converges to the minimum (cid:96) 2 -norm solution with favourable generalization properties. For under-parameterized linear clas-siﬁcation, optimizers can converge to different decision boundaries separating the data. We prove that for any such classiﬁer, there exists a family of quadratic norms (cid:107)·(cid:107) P such that the classiﬁer’s direction is the same as that of the maximum P -margin solution. We argue that analyzing convergence to the standard maximum (cid:96) 2 -margin is arbitrary and show that minimizing the norm induced by the data can result in better generalization. We validate our theoretical results via experiments on synthetic and real datasets.

2019-12-31

(published)

www.semanticscholar.org

Implicit Regularization via Neural Feature Alignment

R Devon Hjelm

We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induc… (see more)ed by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths.

2019-12-31

arXiv (preprint)

G RADIENT -B ASED N EURAL DAG L EARNING WITH I NTERVENTIONS

Alexandre Lacoste

Decision making based on statistical association alone can be a dangerous endeavor due to non-causal associations. Ideally, one would rely o… (see more)n causal relationships that enable reasoning about the effect of interventions. Several methods have been proposed to discover such relationships from observational and inter-ventional data. Among them, GraN-DAG, a method that relies on the constrained optimization of neural networks, was shown to produce state-of-the-art results among algorithms relying purely on observational data. However, it is limited to observational data and cannot make use of interventions. In this work, we extend GraN-DAG to support interventional data and show that this improves its ability to infer causal structures

2019-12-31

(published)

www.semanticscholar.org

Negative Momentum for Improved Game Dynamics

Gauthier Gidel

Reyhane Askari Hemmat

Rémi Le Priol

Games generalize the single-objective optimization paradigm by introducing different objective functions for different players. Differentiab… (see more)le games often proceed by simultaneous or alternating gradient updates. In machine learning, games are gaining new importance through formulations like generative adversarial networks (GANs) and actor-critic systems. However, compared to single-objective optimization, game dynamics are more complex and less understood. In this paper, we analyze gradient-based methods with momentum on simple games. We prove that alternating updates are more stable than simultaneous updates. Next, we show both theoretically and empirically that alternating gradient updates with a negative momentum term achieves convergence in a difficult toy adversarial problem, but also on the notoriously difficult to train saturating GANs.

2019-04-10

International Conference on Artificial Intelligence and Statistics (published)

Predicting Tactical Solutions to Operational Planning Problems under Imperfect Information

Eric Larsen

Andrea Lodi

2018-07-30

arXiv.org (preprint)

arxiv.org

A3T: Adversarially Augmented Adversarial Training

Recent research showed that deep neural networks are highly sensitive to so-called adversarial perturbations, which are tiny perturbations o… (see more)f the input data purposely designed to fool a machine learning classifier. Most classification models, including deep learning models, are highly vulnerable to adversarial attacks. In this work, we investigate a procedure to improve adversarial robustness of deep neural networks through enforcing representation invariance. The idea is to train the classifier jointly with a discriminator attached to one of its hidden layer and trained to filter the adversarial noise. We perform preliminary experiments to test the viability of the approach and to compare it to other standard adversarial training methods.

2018-01-11

ArXiv (preprint)

arxiv.org

Frank-Wolfe Splitting via Augmented Lagrangian Method

Gauthier Gidel

Fabian Pedregosa

Minimizing a function over an intersection of convex sets is an important task in optimization that is often much more challenging than mini… (see more)mizing it over each individual constraint set. While traditional methods such as Frank-Wolfe (FW) or proximal gradient descent assume access to a linear or quadratic oracle on the intersection, splitting techniques take advantage of the structure of each sets, and only require access to the oracle on the individual constraints. In this work, we develop and analyze the Frank-Wolfe Augmented Lagrangian (FW-AL) algorithm, a method for minimizing a smooth function over convex compact sets related by a "linear consistency" constraint that only requires access to a linear minimization oracle over the individual constraints. It is based on the Augmented Lagrangian Method (ALM), also known as Method of Multipliers, but unlike most existing splitting methods, it only requires access to linear (instead of quadratic) minimization oracles. We use recent advances in the analysis of Frank-Wolfe and the alternating direction method of multipliers algorithms to prove a sublinear convergence rate for FW-AL over general convex compact sets and a linear convergence rate for polytopes.

2017-12-31

AISTATS (published)

A Closer Look at Memorization in Deep Networks

Devansh Arpit

Stanisław Jastrzębski

David Krueger

Maxinder S. Kanwal

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While dee… (see more)p networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.

2017-08-05

International Conference on Machine Learning (unknown)