Portrait de Noah Marshall n'est pas disponible

Noah Marshall

Doctorat - McGill
Superviseur⋅e principal⋅e
Adam M. Oberman
Co-supervisor
Courtney Paquette
Sujets de recherche
Apprentissage profond

Publications

Phases of Muon: When Muon Eclipses SignSGD
Elliot Paquette
Noah Marshall
Lucas Benigni
Guangyuan Wang
Atish Agarwala
Courtney Paquette
Recently, Muon and related spectral optimizers have demonstrated strong empirical performance as scalable stochastic methods, often outperfo… (voir plus)rming Adam. Yet their behaviour remains poorly understood. We analyze stochastic spectral optimizers, including Muon, on a high-dimensional matrix-valued least squares problem. We derive explicit deterministic dynamics that provide a tractable framework for studying learning behaviour with a focus on (stochastic) SignSVD, which Muon approximates, and (stochastic) SignSGD, the latter serving as a proxy for Adam. Our analysis shows that for large batch size, SignSVD performs a square-root preconditioning with respect to the data covariance spectrum, while for small batch size smaller eigenmodes behave like SGD, slowing down convergence. We contrast with SignSGD which for generic covariance performs no preconditioning and has no transition, leading to different optimal learning rates and convergence characteristics. The two methods match up to a constant factor with isotropic data, but behave differently with anisotropic data. An analysis of a power law covariance model with data exponent
2026-05-09
arXiv (prépublication)
doi.org
arxiv.org
Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects
Ke Liang Xiao
Noah Marshall
Atish Agarwala
Elliot Paquette
In recent years, signSGD has garnered interest as both a practical optimizer as well as a simple model to understand adaptive optimizers lik… (voir plus)e Adam. Though there is a general consensus that signSGD acts to precondition optimization and reshapes noise, quantitatively understanding these effects in theoretically solvable settings remains difficult. We present an analysis of signSGD in a high dimensional limit, and derive a limiting SDE and ODE to describe the risk. Using this framework we quantify four effects of signSGD: effective learning rate, noise compression, diagonal preconditioning, and gradient noise reshaping. Our analysis is consistent with experimental observations but moves beyond that by quantifying the dependence of these effects on the data and noise distributions. We conclude with a conjecture on how these results might be extended to Adam.
2025-07-14
International Conference on Machine Learning (Accept (poster))
doi.org
proceedings.mlr.press