Simon Lacoste-Julien

aristidebaratin@hotmail.com

Biography

Simon Lacoste-Julien is an associate professor at Mila – Quebec Artificial Intelligence Institute and in the Department of Computer Science and Operations Research (DIRO) at Université de Montréal. He is also a Canada CIFAR AI Chair and heads (part time) the SAIT AI Lab Montréal.

Lacoste-Julien‘s research interests are machine learning and applied mathematics, along with their applications to computer vision and natural language processing. He completed a BSc in mathematics, physics and computer science at McGill University, a PhD in computer science at UC Berkeley and a postdoc at the University of Cambridge.

After spending several years as a researcher at INRIA and the École normale supérieure in Paris, he returned to his home city of Montréal in 2016 to answer Yoshua Bengio’s call to help grow the Montréal AI ecosystem.

Current Students

Reza Babanezhad Harikandeh

Independent visiting researcher - Samsung SAIT

Aristide Baratin

Independent visiting researcher - Samsung SAIT

PhD - Université de Montréal

Collaborating Alumni - Université de Montréal

Principal supervisor :

Kiho Cho

Independent visiting researcher - Samsung SAIT

Marwa El Halabi

Independent visiting researcher - Samsung SAIT

Collaborating Alumni - Université de Montréal

PhD - Université de Montréal

Yash Goyal

Independent visiting researcher - Samsung SAIT

yashgoyal.yg1@gmail.com

Meraj Hashemizadeh

Collaborating researcher - Université de Montréal

Alexia Jolicoeur-Martineau

Independent visiting researcher - Samsung SAIT

PhD - Université de Montréal

Kwon Kisoo

Independent visiting researcher - Seoul National University, Korea

Boris Knyazev

Independent visiting researcher - Université de Montréal

PhD - Université de Montréal

Jaewoo Lee

Independent visiting researcher - Pohang University of Science and Technology in Pohang, Korea

Michelle Liu

Collaborating researcher

Lucas Maes

PhD - Université de Montréal

Master's Research - Université de Montréal

Juan Ramirez

PhD - Université de Montréal

PhD - Université de Montréal

Independent visiting researcher - Samsung SAIT

Motahareh Sohrabi

Collaborating researcher - Université de Montréal

Helen Zhang

PhD - Université de Montréal

Yan Zhang

Independent visiting researcher - Samsung SAIT

Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation

Blog Posts

March 18, 2024

Sébastien Lachapelle

Divyat Mahajan

Ioannis Mitliagkas

Simon Lacoste-Julien

Read the article

Publications

Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity

Nicolas Loizou

Hugo Berard

Gauthier Gidel

Ioannis Mitliagkas

Two of the most prominent algorithms for solving unconstrained smooth games are the classical stochastic gradient descent-ascent (SGDA) and … (see more)the recently introduced stochastic consensus optimization (SCO) [Mescheder et al., 2017]. SGDA is known to converge to a stationary point for specific classes of games, but current convergence analyses require a bounded variance assumption. SCO is used successfully for solving large-scale adversarial problems, but its convergence guarantees are limited to its deterministic variant. In this work, we introduce the expected co-coercivity condition, explain its benefits, and provide the first last-iterate convergence guarantees of SGDA and SCO under this condition for solving a class of stochastic variational inequality problems that are potentially non-monotone. We prove linear convergence of both methods to a neighborhood of the solution when they use constant step-size, and we propose insightful stepsize-switching rules to guarantee convergence to the exact solution. In addition, our convergence guarantees hold under the arbitrary sampling paradigm, and as such, we give insights into the complexity of minibatching.

openreview.net

A Survey of Self-Supervised and Few-Shot Object Detection

Gabriel Huang

Issam Hadj Laradji

David Vazquez

Pau Rodriguez

Labeling data is often expensive and time-consuming, especially for tasks such as object detection and instance segmentation, which require … (see more)dense labeling of the image. While few-shot object detection is about training a model on novel (unseen) object classes with little data, it still requires prior training on many labeled examples of base (seen) classes. On the other hand, self-supervised methods aim at learning representations from unlabeled data which transfer well to downstream tasks such as object detection. Combining few-shot and self-supervised object detection is a promising research direction. In this survey, we review and characterize the most recent approaches on few-shot and self-supervised object detection. Then, we give our main takeaways and discuss future research directions. Project page: https://gabrielhuang.github.io/fsod-survey/.

2021-10-27

ArXiv (preprint)

doi.org

Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence

Nicolas Loizou

Sharan Vaswani

Issam Hadj Laradji

We propose a stochastic variant of the classical Polyak step-size (Polyak, 1987) commonly used in the subgradient method. Although computing… (see more) the Polyak step-size requires knowledge of the optimal function values, this information is readily available for typical modern machine learning applications. Consequently, the proposed stochastic Polyak step-size (SPS) is an attractive choice for setting the learning rate for stochastic gradient descent (SGD). We provide theoretical convergence guarantees for SGD equipped with SPS in different settings, including strongly convex, convex and non-convex functions. Furthermore, our analysis results in novel convergence guarantees for SGD with a constant step-size. We show that SPS is particularly effective when training over-parameterized models capable of interpolating the training data. In this setting, we prove that SPS enables SGD to converge to the true solution at a fast rate without requiring the knowledge of any problem-dependent constants or additional computational overhead. We experimentally validate our theoretical results via extensive experiments on synthetic and real datasets. We demonstrate the strong performance of SGD with SPS compared to state-of-the-art optimization methods when training over-parameterized models.

2021-03-18

Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (published)

proceedings.mlr.press

SVRG meets AdaGrad: painless variance reduction

Benjamin Dubois-Taine

Sharan Vaswani

Reza Babanezhad Harikandeh

Mark Schmidt

2021-02-18

ArXiv (preprint)

doi.org

An Analysis of the Adaptation Speed of Causal Models

Rémi LE PRIOL

Reza Babanezhad Harikandeh

Yoshua Bengio

2021-01-01

AISTATS (published)

proceedings.mlr.press

Implicit Regularization in Deep Learning: A View from Function Space

Aristide Baratin

Thomas George

César Laurent

We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a possible regularization eff… (see more)ect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. By extrapolating a new analysis of Rademacher complexity bounds in linear models, we propose and study a new heuristic complexity measure for neural networks which captures this phenomenon, in terms of sequences of tangent kernel classes along in the learning trajectories.

2020-08-03

ArXiv (preprint)

Implicit Regularization in Deep Learning: A View from Function Space

Aristide Baratin

Thomas George

César Laurent

2020-08-03

ArXiv (preprint)

To Each Optimizer a Norm, To Each Norm its Generalization

Sharan Vaswani

Reza Babanezhad Harikandeh

Jose Gallego

Aaron Mishkin

Nicolas Le Roux

We study the implicit regularization of optimization methods for linear models interpolating the training data in the under-parametrized and… (see more) over-parametrized regimes. Since it is difficult to determine whether an optimizer converges to solutions that minimize a known norm, we flip the problem and investigate what is the corresponding norm minimized by an interpolating solution. Using this reasoning, we prove that for over-parameterized linear regression, projections onto linear spans can be used to move between different interpolating solutions. For under-parameterized linear classification, we prove that for any linear classifier separating the data, there exists a family of quadratic norms ||.||_P such that the classifier's direction is the same as that of the maximum P-margin solution. For linear classification, we argue that analyzing convergence to the standard maximum l2-margin is arbitrary and show that minimizing the norm induced by the data results in better generalization. Furthermore, for over-parameterized linear classification, projections onto the data-span enable us to use techniques from the under-parameterized setting. On the empirical side, we propose techniques to bias optimizers towards better generalizing solutions, improving their test performance. We validate our theoretical results via synthetic experiments, and use the neural tangent kernel to handle non-linear models.

2020-06-11

ArXiv (preprint)

Accelerating Smooth Games by Manipulating Spectral Shapes

Waiss Azizian

Damien Scieur

Ioannis Mitliagkas

Gauthier Gidel

2020-06-03

Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (published)

proceedings.mlr.press

An Analysis of the Adaptation Speed of Causal Models

Rémi LE PRIOL

Reza Babanezhad Harikandeh

Yoshua Bengio

We consider the problem of discovering the causal process that generated a collection of datasets. We assume that all these datasets were ge… (see more)nerated by unknown sparse interventions on a structural causal model (SCM)

2020-05-18

ArXiv (preprint)

Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence

Nicolas Loizou

Sharan Vaswani

Issam Hadj Laradji

2020-02-24

ArXiv (preprint)

Accelerating Smooth Games by Manipulating Spectral Shapes

Waiss Azizian

Damien Scieur

Ioannis Mitliagkas

Gauthier Gidel

We use matrix iteration theory to characterize acceleration in smooth games. We define the spectral shape of a family of games as the set co… (see more)ntaining all eigenvalues of the Jacobians of standard gradient dynamics in the family. Shapes restricted to the real line represent well-understood classes of problems, like minimization. Shapes spanning the complex plane capture the added numerical challenges in solving smooth games. In this framework, we describe gradient-based methods, such as extragradient, as transformations on the spectral shape. Using this perspective, we propose an optimal algorithm for bilinear games. For smooth and strongly monotone operators, we identify a continuum between convex minimization, where acceleration is possible using Polyak's momentum, and the worst case where gradient descent is optimal. Finally, going beyond first-order methods, we propose an accelerated version of consensus optimization.

2020-01-02

ArXiv (preprint)