Portrait of Elliot Paquette

Elliot Paquette

Associate Academic Member
Associate Professor, McGill University, Department of Mathematics and Statistics
Research Topics
Machine Learning Theory
Optimization
Website
Google Scholar
X

Publications

Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions
Kiwon Lee
Andrew N. Cheng
Courtney Paquette
Elliot Paquette
We analyze the dynamics of large batch stochastic gradient descent with momentum (SGD+M) on the least squares problem when both the number o… (see more)f samples and dimensions are large. In this setting, we show that the dynamics of SGD+M converge to a deterministic discrete Volterra equation as dimension increases, which we analyze. We identify a stability measurement, the implicit conditioning ratio (ICR), which regulates the ability of SGD+M to accelerate the algorithm. When the batch size exceeds this ICR, SGD+M converges linearly at a rate of
2021-12-31
Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (published)
doi.org
openreview.net