Portrait de Pietro Cagnasso

Pietro Cagnasso

Doctorat - Concordia
Superviseur⋅e principal⋅e
Eugene Belilovsky
Co-supervisor
Irina Rish
Sujets de recherche
Apprentissage profond
Site web
GitHub

Publications

Unifying Local Communications and Local Updates for LLM Pretraining
Pietro Cagnasso
Eugene Belilovsky
Edouard Oyallon
Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data center… (voir plus)s, and lower-bandwidth links. Many practical methods reduce communication frequency but still rely on synchronous All-Reduce operations that maintain identical model states and tie progress to global collectives. This can become a bottleneck when bandwidth or worker speed is heterogeneous. We introduce GASLoC, a novel decentralized pre-training algorithm that generalizes the notion of communication acceleration to the recently popular"outer optimizer"to allow a practical gossip-based training framework that is compatible with adaptive optimizers, allows for local optimizer steps, and can utilize sparse randomized peer communication. Empirically, on a number of standard LLM training tasks, we demonstrate that GASLoC outperforms state-of-the-art decentralized algorithms in single step per communication setting for a number of topologies and, unlike existing decentralized methods in the LLM setting, it allows to obtain performance competitive with DiLoCo when utilizing multiple local steps. In the heterogeneous bandwidth setting we demonstrate the advantage of GASLoC showing that it can significantly outperform DiLoCo.
2026-06-08
arXiv (prépublication)
doi.org
arxiv.org