Portrait de Mahdi Ghaznavi

Mahdi Ghaznavi

mahdi.ghaznavi@mila.quebec
Collaborateur·rice de recherche
Superviseur⋅e principal⋅e
Ioannis Mitliagkas
Sujets de recherche
Apprentissage de représentations
Apprentissage profond
Généralisation
Généralisation hors distribution (OOD)
IAG (Intelligence Artificielle Générale)
Modèles de fondation
Optimisation
Google Scholar
GitHub

Publications

Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Hiroki Naganuma
Laura Gomezjurado
Mahdi Ghaznavi
Ganesh Talluri
Atsushi Nitanda
Seng Pei Liew
Ryuichiro Hataya
Ioannis Mitliagkas
Modern neural-network training uses mixed recipes, assigning AdamW-like or sign-based updates to embedding and output layers and spectral up… (voir plus)dates to hidden matrices. These assignments are usually fixed by layer type, without a clear criterion for selecting the appropriate geometry. We turn sign-versus-spectral assignment into a measurable, layerwise decision problem. For ideal memoryless steps, we derive a finite-batch ratio of conservative descent-lemma lower bounds for spectral and sign updates. Its closed form separates signal efficiency from a geometry-dependent global curvature factor. In the positive-signal regime, the same smoothness assumption gives two-sided bounds that determine which step has the larger conditional expected decrease when the ratio crosses explicit noise-dependent thresholds. We then derive a local directional quadratic score from gradient alignments and Hessian-vector products and use it in Generalized-Scion (G-Scion) to control layerwise routing. In a controlled input-rank experiment, including directional curvature matches the measured same-iterate ordering in all seven aggregate settings, whereas the curvature-free proxy selects the opposite geometry. On 31M GPT-2, the output score routes the output layer from spectral warmup to sign, and G-Scion improves validation loss over fixed Scion in memoryless and matched-momentum comparisons. ViT-B/16 results further show that a spectral edge assignment can outperform fixed Scion outside language modelling. These results suggest that optimizer geometry need not be hard-coded by layer type and are a first step toward understanding how it should be assigned.
2026-10-11
arXiv (prépublication)
doi.org
arxiv.org
The Geometry of Spectral Gradient Descent: Layerwise Criteria for SignSGD vs SpecSGD
Hiroki Naganuma
Laura Gomezjurado
Mahdi Ghaznavi
Ioannis Mitliagkas
Optimization in deep learning has expanded beyond Euclidean methods to include entrywise sign updates (SignSGD) and spectral sign updates (S… (voir plus)pecGD/Muon). While both can be viewed as steepest descent under non-Euclidean geometries (
2026-03-01
GRaM @ International Conference on Learning Representations (poster)
openreview.net
openreview.net