Portrait of Ioannis Mitliagkas

Ioannis Mitliagkas

Core Academic Member
ioannis@mila.quebec
Canada CIFAR AI Chair
Associate Professor, Université de Montréal, Department of Computer Science and Operations Research
Research Scientist, Google DeepMind
Research Topics
Deep Learning
Distributed Systems
Dynamical Systems
Generative Models
Machine Learning Theory
Optimization
Representation Learning
Website
Google Scholar
LinkedIn
X

Biography

Ioannis Mitliagkas is an associate professor in the Department of Computer Science and Operations Research (DIRO) at Université de Montréal, as well as a Core Academic member of Mila – Quebec Artificial Intelligence Institute and a Canada CIFAR AI Chair. I hold a part-time position as a staff research scientist at Google DeepMind Montréal.

Previously, I was a postdoctoral scholar in the departments of statistics and computer science at Stanford University. I obtained my PhD from the Department of Electrical and Computer Engineering at the University of Texas at Austin.

My research interests lie in statistical learning and inference, with a focus on optimization, efficient large-scale and distributed algorithms, statistical learning theory and MCMC methods. My recent research has focused on methods for efficient and adaptive optimization, understanding the interaction between optimization and the dynamics of large-scale learning systems, and the dynamics of games.

Current Students

Mehdi Inane Ahmed
PhD - Université de Montréal
Github
Google Scholar
Ange-Clément Akazan
Research Intern - Université de Montréal
Reyhane Askari Hemmat
PhD - Université de Montréal
Co-supervisor :
Nicolas Le Roux
reyhane.askari.hemmat@mila.quebec
Oscar Clivio
Research Intern - Université de Montréal
Co-supervisor :
Alexandre Drouin
Website
Github
Google Scholar
Ryan D'Orazio
PhD - Université de Montréal
Website
Google Scholar
Kilian Fatras
Postdoctorate - McGill University
Principal supervisor :
Adam M. Oberman
Website
Github
Google Scholar
Charles Guille-escuret
PhD - Université de Montréal
Github
Mehrab Hamidi
PhD - Université de Montréal
Github
Google Scholar
Adam Ibrahim
PhD - Université de Montréal
Co-supervisor :
Irina Rish
Website
Google Scholar
Zichu Liu
PhD - Université de Montréal
Principal supervisor :
Gauthier Gidel
Divyat Mahajan
PhD - Université de Montréal
Mehrnaz Mofakhami
Master's Research - Université de Montréal
Principal supervisor :
Gauthier Gidel
Website
Github
Google Scholar
Hiroki Naganuma
PhD - Université de Montréal
Website
Google Scholar
Vincent Quirion
Master's Research - Université de Montréal
Github

Publications

Negative Momentum for Improved Game Dynamics
Gauthier Gidel
Reyhane Askari Hemmat
Mohammad Pezeshki
Gabriel Huang
Rémi LE PRIOL
Simon Lacoste-Julien
Ioannis Mitliagkas
Games generalize the single-objective optimization paradigm by introducing different objective functions for different players. Differentiab… (see more)le games often proceed by simultaneous or alternating gradient updates. In machine learning, games are gaining new importance through formulations like generative adversarial networks (GANs) and actor-critic systems. However, compared to single-objective optimization, game dynamics are more complex and less understood. In this paper, we analyze gradient-based methods with momentum on simple games. We prove that alternating updates are more stable than simultaneous updates. Next, we show both theoretically and empirically that alternating gradient updates with a negative momentum term achieves convergence in a difficult toy adversarial problem, but also on the notoriously difficult to train saturating GANs.
2019-04-11
Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics (published)
proceedings.mlr.press
arxiv.org
Multi-objective training of Generative Adversarial Networks with multiple discriminators
Isabela Albuquerque
Joao Monteiro
Thang Doan
Breandan Considine
T. Falk
Ioannis Mitliagkas
Recent literature has demonstrated promising results for training Generative Adversarial Networks by employing a set of discriminators, in c… (see more)ontrast to the traditional game involving one generator against a single adversary. Such methods perform single-objective optimization on some simple consolidation of the losses, e.g. an arithmetic average. In this work, we revisit the multiple-discriminator setting by framing the simultaneous minimization of losses provided by different models as a multi-objective optimization problem. Specifically, we evaluate the performance of multiple gradient descent and the hypervolume maximization algorithm on a number of different datasets. Moreover, we argue that the previously proposed methods and hypervolume maximization can all be seen as variations of multiple gradient descent in which the update direction can be computed efficiently. Our results indicate that hypervolume maximization presents a better compromise between sample quality and computational cost than previous methods.
2019-01-01
ICML (published)
arxiv.org
arxiv.org
Reducing the variance in online optimization by transporting past gradients
Sébastien M. R. Arnold
Pierre-Antoine Manzagol
Reza Babanezhad Harikandeh
Ioannis Mitliagkas
Nicolas Le Roux
Most stochastic optimization methods use gradients once before discarding them. While variance reduction methods have shown that reusing pas… (see more)t gradients can be beneficial when there is a finite number of datapoints, they do not easily extend to the online setting. One issue is the staleness due to using past gradients. We propose to correct this staleness using the idea of implicit gradient transport (IGT) which transforms gradients computed at previous iterates into gradients evaluated at the current iterate without using the Hessian explicitly. In addition to reducing the variance and bias of our updates over time, IGT can be used as a drop-in replacement for the gradient estimate in a number of well-understood methods such as heavy ball or Adam. We show experimentally that it achieves state-of-the-art results on a wide range of architectures and benchmarks. Additionally, the IGT gradient estimator yields the optimal asymptotic convergence rate for online stochastic optimization in the restricted setting where the Hessians of all component functions are equal.
arxiv.org
Negative Momentum for Improved Game Dynamics
Gauthier Gidel
Reyhane Askari Hemmat
Mohammad Pezeshki
Gabriel Huang
Rémi LE PRIOL
Simon Lacoste-Julien
Ioannis Mitliagkas
Games generalize the single-objective optimization paradigm by introducing different objective functions for different players. Differentiab… (see more)le games often proceed by simultaneous or alternating gradient updates. In machine learning, games are gaining new importance through formulations like generative adversarial networks (GANs) and actor-critic systems. However, compared to single-objective optimization, game dynamics are more complex and less understood. In this paper, we analyze gradient-based methods with momentum on simple games. We prove that alternating updates are more stable than simultaneous updates. Next, we show both theoretically and empirically that alternating gradient updates with a negative momentum term achieves convergence in a difficult toy adversarial problem, but also on the notoriously difficult to train saturating GANs.
2018-07-12
ArXiv (preprint)
arxiv.org
arxiv.org
Accelerated Stochastic Power Iteration
Peng Xu
Bryan Dawei He
Christopher De Sa
Ioannis Mitliagkas
Christopher Re
Principal component analysis (PCA) is one of the most powerful tools in machine learning. The simplest method for PCA, the power iteration, … (see more)requires O ( 1 / Δ ) full-data passes to recover the principal component of a matrix with eigen-gap Δ. Lanczos, a significantly more complex method, achieves an accelerated rate of O ( 1 / Δ ) passes. Modern applications, however, motivate methods that only ingest a subset of available data, known as the stochastic setting. In the online stochastic setting, simple algorithms like Oja's iteration achieve the optimal sample complexity O ( σ 2 / Δ 2 ) . Unfortunately, they are fully sequential, and also require O ( σ 2 / Δ 2 ) iterations, far from the O ( 1 / Δ ) rate of Lanczos. We propose a simple variant of the power iteration with an added momentum term, that achieves both the optimal sample and iteration complexity. In the full-pass setting, standard analysis shows that momentum achieves the accelerated rate, O ( 1 / Δ ) . We demonstrate empirically that naively applying momentum to a stochastic method, does not result in acceleration. We perform a novel, tight variance analysis that reveals the "breaking-point variance" beyond which this acceleration does not occur. By combining this insight with modern variance reduction techniques, we construct stochastic PCA algorithms, for the online and offline setting, that achieve an accelerated iteration complexity O ( 1 / Δ ) . Due to the embarassingly parallel nature of our methods, this acceleration translates directly to wall-clock time if deployed in a parallel environment. Our approach is very general, and applies to many non-convex optimization problems that can now be accelerated using the same technique.
2018-01-01
AISTATS (published)
proceedings.mlr.press
arxiv.org
Deep Learning @15 Petaflops/second: Semi-supervised pattern detection for 15 Terabytes of climate data
W. Collins
M. Wehner
M. Prabhat
Thorsten Kurth
Nadathur Satish
Ioannis Mitliagkas
Jian Zhang
Evan Racah
Md. Mostofa Ali Patwary
Narayanan Sundaram
Pradeep Dubey
2017-12-01
(published)
www.semanticscholar.org
Accelerated Stochastic Power Iteration
Peng Xu
Bryan Dawei He
Christopher De Sa
Ioannis Mitliagkas
Christopher Re
Principal component analysis (PCA) is one of the most powerful tools in machine learning. The simplest method for PCA, the power iteration, … (see more)requires O ( 1 / Δ ) full-data passes to recover the principal component of a matrix with eigen-gap Δ. Lanczos, a significantly more complex method, achieves an accelerated rate of O ( 1 / Δ ) passes. Modern applications, however, motivate methods that only ingest a subset of available data, known as the stochastic setting. In the online stochastic setting, simple algorithms like Oja's iteration achieve the optimal sample complexity O ( σ 2 / Δ 2 ) . Unfortunately, they are fully sequential, and also require O ( σ 2 / Δ 2 ) iterations, far from the O ( 1 / Δ ) rate of Lanczos. We propose a simple variant of the power iteration with an added momentum term, that achieves both the optimal sample and iteration complexity. In the full-pass setting, standard analysis shows that momentum achieves the accelerated rate, O ( 1 / Δ ) . We demonstrate empirically that naively applying momentum to a stochastic method, does not result in acceleration. We perform a novel, tight variance analysis that reveals the "breaking-point variance" beyond which this acceleration does not occur. By combining this insight with modern variance reduction techniques, we construct stochastic PCA algorithms, for the online and offline setting, that achieve an accelerated iteration complexity O ( 1 / Δ ) . Due to the embarassingly parallel nature of our methods, this acceleration translates directly to wall-clock time if deployed in a parallel environment. Our approach is very general, and applies to many non-convex optimization problems that can now be accelerated using the same technique.
2017-07-10
ArXiv (preprint)
arxiv.org
arxiv.org