Portrait de Pietro Cagnasso

Pietro Cagnasso

pietro.cagnasso@mila.quebec
Doctorat - Concordia
Superviseur⋅e principal⋅e
Eugene Belilovsky
Co-supervisor
Irina Rish
Sujets de recherche
Apprentissage profond
Site web
GitHub

Publications

Unifying Local Communications and Local Updates for LLM Pretraining
Pietro Cagnasso
Eugene Belilovsky
Edouard Oyallon
Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data center… (voir plus)s, and lower-bandwidth links. Many practical methods reduce communication frequency but still rely on synchronous All-Reduce operations that maintain identical model states and tie progress to global collectives. This can become a bottleneck when bandwidth or worker speed is heterogeneous. We introduce GASLoC, a novel decentralized pre-training algorithm that generalizes the notion of communication acceleration to the recently popular"outer optimizer"to allow a practical gossip-based training framework that is compatible with adaptive optimizers, allows for local optimizer steps, and can utilize sparse randomized peer communication. Empirically, on a number of standard LLM training tasks, we demonstrate that GASLoC outperforms state-of-the-art decentralized algorithms in single step per communication setting for a number of topologies and, unlike existing decentralized methods in the LLM setting, it allows to obtain performance competitive with DiLoCo when utilizing multiple local steps. In the heterogeneous bandwidth setting we demonstrate the advantage of GASLoC showing that it can significantly outperform DiLoCo.
2026-06-08
arXiv (prépublication)
doi.org
arxiv.org
Model Parallelism With Subnetwork Data Parallelism
Vaibhav Singh
Zafir Khalid
Pietro Cagnasso
Edouard Oyallon
Eugene Belilovsky
Distributed pre-training of large models at scale often imposes heavy memory demands on individual nodes and incurs significant intra-node c… (voir plus)ommunication costs. We propose a novel alternative approach that reduces the memory requirements by training small, structured subnetworks of the model on separate workers. Unlike pipelining, our method avoids inter-node activation communication and maintains bandwidth requirements that are comparable to or lower than standard data parallel communication schemes based on all-reduce. We evaluate two subnetwork construction strategies guided by the principle of ensuring uniform representation of each parameter across the distributed training setup. Our results show that the stochastic block dropping technique consistently outperforms the width-wise subnetwork construction previously explored in federated learning. We empirically attribute this superior performance to stronger gradient alignment in subnetworks that retain blocks having skip connections. Preliminary experiments highlight the promise of our approach, achieving a
2025-06-10
ES-FoMo @ International Conference on Machine Learning (publié)
doi.org
openreview.net