Portrait de Xiaolong Huang

Xiaolong Huang

xiaolong.huang@mila.quebec
Maîtrise recherche - Concordia
Superviseur⋅e principal⋅e
Eugene Belilovsky
Co-supervisor
Guy Wolf
Sujets de recherche
Apprentissage de représentations
Apprentissage multimodal
Optimisation
Vision par ordinateur
Site web
Google Scholar
GitHub

Publications

Efficient Long-Horizon Learning for Learned Optimization
Xiaolong Huang
Benjamin Therien
James Harrison
Eugene Belilovsky
Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers ov… (voir plus)er a distribution of tasks. While recent work has greatly advanced the architectural design and inductive biases of learned optimizers (LOs), current meta-training approaches still suffer from two main difficulties: (1) they cannot efficiently scale meta-training to long-horizon inner problems and (2) they often fail to compete with strong hand-designed optimizers. To address these limitations, we propose Efficient Long-hOrizon (ELO) learning, an efficient meta-training algorithm that (1) reallocates redundant meta-training compute to longer failure regimes, achieving efficient long-horizon learning, and (2) enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally improve the generalization of LOs. Our empirical study evaluates ELO for meta-training both element-wise and matrix-based LOs. Across downstream language modeling (GPT-2-124M/350M on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K) tasks, ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs. In particular, ELO-Celo2 consistently outperforms well-tuned AdamW across all evaluated tasks, while remaining competitive with Muon on language modeling. \textit{Notably, all ELO baselines require less than 7 H100 GPU-hours for meta-training.}
2026-07-06
arXiv (prépublication)
doi.org
arxiv.org
Empowering 2D neural network for 3D medical image segmentation via neighborhood information fusion
Qiankun Li
Xiaolong Huang
Yani Zhang
Bo Fang
Duo Hong
Junxin Chen
2026-06-30
Pattern Recognition (publié)
doi.org
MuLoCo: Muon is a practical inner optimizer for DiLoCo
Benjamin Therien
Xiaolong Huang
Irina Rish
Eugene Belilovsky
2025-06-10
ICML.cc/2025/Workshop/ES-FoMo-III (publié)
doi.org
openreview.net
PyLO: Towards Accessible Learned Optimizers in PyTorch
Paul Janson
Benjamin Therien
Quentin Gregory Anthony
Xiaolong Huang
Abhinav Moudgil
Eugene Belilovsky
Learned optimizers have been an active research topic over the past decade, with increasing progress toward practical, general-purpose optim… (voir plus)izers that can serve as drop-in replacements for widely used methods like Adam. However, recent advances -- such as VeLO, which was meta-trained for 4000 TPU-months -- remain largely inaccessible to the broader community, in part due to their reliance on JAX and the absence of user-friendly packages for applying the optimizers after meta-training. To address this gap, we introduce PyLO, a PyTorch-based library that brings learned optimizers to the broader machine learning community through familiar, widely adopted workflows. Unlike prior work focused on synthetic or convex tasks, our emphasis is on applying learned optimization to real-world large-scale pre-training tasks. Our release includes a CUDA-accelerated version of the small_fc_lopt learned optimizer architecture from (Metz et al., 2022a), delivering substantial speedups -- from 39.36 to 205.59 samples/sec throughput for training ViT B/16 with batch size 32. PyLO also allows us to easily combine learned optimizers with existing optimization tools such as learning rate schedules and weight decay. When doing so, we find that learned optimizers can substantially benefit. Our code is available at https://github.com/Belilovsky-Lab/pylo
2025-06-08
ICML.cc/2025/Workshop/CODEML (publié)
doi.org
openreview.net