Portrait de Elliot Paquette

Elliot Paquette

Membre académique associé
Professeur agrégé, McGill University, Département de mathématiques et de statistique
Sujets de recherche
Optimisation
Théorie de l'apprentissage automatique
Site web
Google Scholar
X

Publications

Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions
Kiwon Lee
Andrew N. Cheng
Courtney Paquette
Elliot Paquette
We analyze the dynamics of large batch stochastic gradient descent with momentum (SGD+M) on the least squares problem when both the number o… (voir plus)f samples and dimensions are large. In this setting, we show that the dynamics of SGD+M converge to a deterministic discrete Volterra equation as dimension increases, which we analyze. We identify a stability measurement, the implicit conditioning ratio (ICR), which regulates the ability of SGD+M to accelerate the algorithm. When the batch size exceeds this ICR, SGD+M converges linearly at a rate of
2021-12-31
Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (publié)
doi.org
openreview.net