Portrait de Zafir Khalid n'est pas disponible

Zafir Khalid

Maîtrise recherche - Concordia
Superviseur⋅e principal⋅e
Eugene Belilovsky
Co-supervisor
Irina Rish
Sujets de recherche
Apprentissage fédéré
Apprentissage profond
Vision par ordinateur
Site web
GitHub

Publications

Model Parallelism With Subnetwork Data Parallelism
Vaibhav Singh
Zafir Khalid
Edouard Oyallon
Eugene Belilovsky
Pre-training large neural networks at scale imposes heavy memory demands on accelerators and often requires costly communication. We introdu… (voir plus)ce Subnetwork Data Parallelism (SDP), a distributed training framework that partitions a model into structured subnetworks trained across workers without exchanging activations. We study two complementary masking regimes: backward masking, which applies sparsity only in the backward step to retain unbiased gradients, and forward masking, which also removes parameters in the forward pass to deliver stronger efficiency gains while providing additional regularization. We further explore two subnetwork construction strategies: neuron level and block level, applied across both CNNs and transformers. In experiments spanning CNNs and transformers on CIFAR and ImageNet, as well as LLM pre-training on FineWeb, SDP reduces per-device memory usage by 30%-75% while maintaining or improving performance. Notably, in FLOP-matched settings, forward masking can sometimes achieve better performance.
2025-07-11
ArXiv (prépublication)
doi.org
arxiv.org
Model Parallelism With Subnetwork Data Parallelism
Vaibhav Singh
Zafir Khalid
Edouard Oyallon
Eugene Belilovsky
Distributed pre-training of large models at scale often imposes heavy memory demands on individual nodes and incurs significant intra-node c… (voir plus)ommunication costs. We propose a novel alternative approach that reduces the memory requirements by training small, structured subnetworks of the model on separate workers. Unlike pipelining, our method avoids inter-node activation communication and maintains bandwidth requirements that are comparable to or lower than standard data parallel communication schemes based on all-reduce. We evaluate two subnetwork construction strategies guided by the principle of ensuring uniform representation of each parameter across the distributed training setup. Our results show that the stochastic block dropping technique consistently outperforms the width-wise subnetwork construction previously explored in federated learning. We empirically attribute this superior performance to stronger gradient alignment in subnetworks that retain blocks having skip connections. Preliminary experiments highlight the promise of our approach, achieving a
2025-06-11
ICML.cc/2025/Workshop/ES-FoMo-III (publié)
doi.org
openreview.net