Hiroki Naganuma

Generalization Measures under Controlled Covariate Shift: A Regime-Aware Benchmark

Sora Nakai

Youssef Fadhloun

Kacem Mathlouthi

Kotaro Yoshida

Ganesh Talluri

Predicting generalization from quantities available before target-test evaluation remains a central challenge in deep learning. The systemat… (voir plus)ic benchmark of Jiang et al. (2020) evaluated many generalization measures, but it focused on independent and identically distributed (IID) settings. We revisit this problem for image classifiers evaluated under controlled corruptions and perturbations. Our study uses CIFAR-10-C/P, where the label space and task remain fixed while the input images are degraded or perturbed. This setting also allows us to revisit the robustness concerns raised by Dziugaite et al. (2020), who showed that the apparent reliability of generalization measures can depend strongly on experimental conditions. Our experiments show that the usefulness of generalization measures is strongly regime-dependent. In our exploratory decision analysis across three CNN-style architectures, sharpness- and input-gradient-based measures are among the leading individual signals, whereas family results are close and architecture dependent. Optimization-based measures, Information Criteria, and Sharpness-based measures provide additional regime-dependent signals in correlation or local-reliability analyses. Together, these findings suggest that model selection should not rely only on measures favored by IID evaluation. Instead, within the evaluated CIFAR-10-C/P setting and architectures, generalization measures should be treated as regime-dependent ranking signals whose utility must be evaluated for the intended corruption or perturbation setting.

2026-08-24

Transactions on Machine Learning Research (accepté)

doi.org

openreview.net

Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization

Tatsuhiro Nakamori

Laura Gomezjurado Gonzalez

Ganesh Talluri

Ansh Tiwari

Hideyuki Kawashima

Ioannis Mitliagkas

Guillaume Rabusseau

Hiroki Naganuma

Low-rank gradient compression reduces communication in distributed training by representing updates with rank-…

2026-05-28

CoLoRAI @ International Conference on Machine Learning (poster)

doi.org

openreview.net

Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent

Hiroki Naganuma

Shagun Gupta

Youssef Briki

Ioannis Mitliagkas

Irina Rish

Parameswaran Raman

Hao-Jun Michael Shi

To maximize hardware utilization, modern machine learning systems typically employ large constant or manually tuned batch size schedules, re… (voir plus)lying on heuristics that are brittle and costly to tune. Existing adaptive strategies based on gradient noise scale (GNS) offer a principled alternative. However, their assumption of SGD's Euclidean geometry creates a fundamental mismatch with popular optimizers based on generalized norms, such as signSGD / Signum (

2026-04-29

International Conference on Machine Learning (accepté)

doi.org

openreview.net

The Geometry of Spectral Gradient Descent: Layerwise Criteria for SignSGD vs SpecSGD

Hiroki Naganuma

Laura Gomezjurado

Mahdi Ghaznavi

Ioannis Mitliagkas

Optimization in deep learning has expanded beyond Euclidean methods to include entrywise sign updates (SignSGD) and spectral sign updates (S… (voir plus)pecGD/Muon). While both can be viewed as steepest descent under non-Euclidean geometries (

2026-03-01

GRaM @ International Conference on Learning Representations (poster)

openreview.net

Revisiting Generalization Measures Beyond IID: An Empirical Study under Distributional Shift

Sora Nakai

Youssef Fadhloun

Kacem Mathlouthi

Kotaro Yoshida

Ganesh Talluri

Ioannis Mitliagkas

Hiroki Naganuma

Generalization remains a central yet unresolved challenge in deep learning, particularly the ability to predict a model's performance beyond… (voir plus) its training distribution using quantities available prior to test-time evaluation. Building on the large-scale study of Jiang et al. (2020). and concerns by Dziugaite et al. (2020). about instability across training configurations, we benchmark the robustness of generalization measures beyond IID regime. We train small-to-medium models over 10,000 hyperparameter configurations and evaluate more than 40 measures computable from the trained model and the available training data alone. We significantly broaden the experimental scope along multiple axes: (i) extending the evaluation beyond the standard IID setting to include benchmarking for robustness across diverse distribution shifts, (ii) evaluating multiple architectures and training recipes, and (iii) newly incorporating calibration- and information-criteria-based measures to assess their alignment with both IID and OOD generalization. We find that distribution shifts can substantially alter the predictive performance of many generalization measures, while a smaller subset remains comparatively stable across settings.

2026-02-01

Open MIND (prépublication)

doi.org

arxiv.org

DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging

Kotaro Yoshida

Yuji Naraki

Takafumi Horie

Ryotaro Shimizu

Ioannis Mitliagkas

Hiroki Naganuma

Model merging has emerged as an efficient and flexible paradigm for multi-task learning, with numerous methods being proposed in recent year… (voir plus)s. However, these state-of-the-art techniques are typically evaluated on benchmark suites that are highly favorable to model merging, and their robustness in more realistic settings remains largely unexplored. In this work, we first investigate the vulnerabilities of model-merging methods and pinpoint the source-model characteristics that critically underlie them. Specifically, we identify two factors that are particularly harmful to the merging process: (1) disparities in task vector norms, and (2) the low confidence of the source models. To address this issue, we propose **DisTaC** (**Dis**tillation for **Ta**sk vector **C**onditioning), a novel method that pre-conditions these problematic task vectors before the merge. DisTaC leverages knowledge distillation to adjust a task vector's norm and increase source-model confidence while preserving its essential task-specific knowledge. Our extensive experiments demonstrate that by pre-conditioning task vectors with DisTaC, state-of-the-art merging techniques can successfully integrate models that exhibit these harmful traits, where they would otherwise fail, and achieve significant performance gains.

2025-12-31

International Conference on Learning Representations (Accept (Poster))

doi.org

openreview.net

Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training

Hiroki Naganuma

Xinzhi Zhang

Man-Chung Yue

Ioannis Mitliagkas

Russell J. Hewett

Philipp Andre Witte

Yin Tat Lee

2025-09-14

TMLR (accepté)

openreview.net

An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration

Hiroki Naganuma

Ryuichiro Hataya

Kotaro Yoshida

Ioannis Mitliagkas

2025-05-12

TMLR (accepté)

openreview.net

No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths

Charles Guille-escuret

Hiroki Naganuma

Kilian Fatras

Ioannis Mitliagkas

Understanding the optimization dynamics of neural networks is necessary for closing the gap between theory and practice. Stochastic first-or… (voir plus)der optimization algorithms are known to efficiently locate favorable minima in deep neural networks. This efficiency, however, contrasts with the non-convex and seemingly complex structure of neural loss landscapes. In this study, we delve into the fundamental geometric properties of sampled gradients along optimization paths. We focus on two key quantities, which appear in the restricted secant inequality and error bound. Both hold high significance for first-order optimization. Our analysis reveals that these quantities exhibit predictable, consistent behavior throughout training, despite the stochasticity induced by sampling minibatches. Our findings suggest that not only do optimization trajectories never encounter significant obstacles, but they also maintain stable dynamics during the majority of training. These observed properties are sufficiently expressive to theoretically guarantee linear convergence and prescribe learning rate schedules mirroring empirical practices. We conduct our experiments on image classification, semantic segmentation and language modeling across different batch sizes, network architectures, datasets, optimizers, and initialization seeds. We discuss the impact of each factor. Our work provides novel insights into the properties of neural network loss functions, and opens the door to theoretical frameworks more relevant to prevalent practice.

2024-07-22

International Conference on Machine Learning (Accept (Poster))

doi.org

proceedings.mlr.press

Smoothness-Adaptive Sharpness-Aware Minimization for Finding Flatter Minima

Hiroki Naganuma

Junhyung Lyle Kim

Anastasios Kyrillidis

Ioannis Mitliagkas

The sharpness-aware minimization (SAM) procedure recently gained increasing attention due to its favorable generalization ability to unseen … (voir plus)data. SAM aims to find flatter (local) minima, utilizing a minimax objective. An immediate challenge in the application of SAM is the adjustment of two pivotal step sizes, which significantly influence its effectiveness. We introduce a novel, straightforward approach for adjusting step sizes that adapts to the smoothness of the objective function, thereby reducing the necessity for manual tuning. This method, termed Smoothness-Adaptive SAM (SA-SAM), not only simplifies the optimization process but also promotes the method's inherent tendency to converge towards flatter minima, enhancing performance in specific models.

2024-03-04

ICLR.cc/2024/Workshop/PML4LRS (poster)

openreview.net

Empirical Study on Optimizer Selection for Out-of-Distribution Generalization

Hiroki Naganuma

Kartik Ahuja

Ioannis Mitliagkas

Shiro Takagi

Tetsuya Motokawa

Rio Yokota

Kohta Ishikawa

Ikuro Sato

Modern deep learning systems do not generalize well when the test data distribution is slightly different to the training data distribution.… (voir plus) While much promising work has been accomplished to address this fragility, a systematic study of the role of optimizers and their out-of-distribution generalization performance has not been undertaken. In this study, we examine the performance of popular first-order optimizers for different classes of distributional shift under empirical risk minimization and invariant risk minimization. We address this question for image and text classification using DomainBed, WILDS, and Backgrounds Challenge as testbeds for studying different types of shifts---namely correlation and diversity shift. We search over a wide range of hyperparameters and examine classification accuracy (in-distribution and out-of-distribution) for over 20,000 models. We arrive at the following findings, which we expect to be helpful for practitioners: i) adaptive optimizers (e.g., Adam) perform worse than non-adaptive optimizers (e.g., SGD, momentum SGD) on out-of-distribution performance. In particular, even though there is no significant difference in in-distribution performance, we show a measurable difference in out-of-distribution performance. ii) in-distribution performance and out-of-distribution performance exhibit three types of behavior depending on the dataset---linear returns, increasing returns, and diminishing returns. For example, in the training of natural language data using Adam, fine-tuning the performance of in-distribution performance does not significantly contribute to the out-of-distribution generalization performance.

2023-06-15

TMLR (accepté)

doi.org

openreview.net

La plateforme Mila Ventures

Boussole des politiques en IA

Séances d'information virtuelles

Hiroki Naganuma

Publications

La plateforme Mila Ventures

Boussole des politiques en IA

Séances d'information virtuelles

Mots-clés populaires:

Hiroki Naganuma

Publications