Charles-Etienne Joseph

Communication-efficient variants of SGD, specifically local SGD, have received a great deal of interest in recent years. These approaches co… (see more)mpute multiple gradient steps locally, that is on each worker, before averaging model parameters, helping relieve the critical communication bottleneck in distributed deep learning training. Although many variants of these approaches have been proposed, they can sometimes lag behind state-of-the-art optimizers for deep learning. In this work, we incorporate local optimizers that compute multiple updates into a learned optimization framework, allowing to meta-learn potentially more efficient local SGD algorithms. Our results demonstrate that local learned optimizers can substantially outperform local SGD and its sophisticated variants while maintaining their communication efficiency. We show that the learned optimizers can generalize to new datasets and architectures, demonstrating the potential of learned optimizers for improving communication-efficient distributed learning.

2000-01-01

(published)

www.semanticscholar.org

Hugo Larochelle appointed Scientific Director of Mila

Leveraging AI for a Sustainable Future

Mila AI Policy Fellowship

Charles-Etienne Joseph

Publications

Hugo Larochelle appointed Scientific Director of Mila

Leveraging AI for a Sustainable Future

Mila AI Policy Fellowship

Popular keywords:

Charles-Etienne Joseph

Publications