Gradient compression is a popular technique for improving communication complexity of
stochastic first-order methods in distributed training of machine learning models. However …

Inspired by a recent breakthrough of Mishchenko et al.[2022], who for the first time showed
that local gradient steps can lead to provable communication acceleration, we propose an …

Federated Learning is a collaborative training framework that leverages heterogeneous data
distributed across a vast number of clients. Since it is practically infeasible to request and …

The backtracking line-search is an effective technique to automatically tune the step-size in
smooth optimization. It guarantees similar performance to using the theoretically optimal …

Communication compression is a crucial technique for modern distributed learning systems
to alleviate their communication bottlenecks over slower networks. Despite recent intensive …

Abstract Knowledge distillation is a popular approach for enhancing the performance of"
student" models, with lower representational capacity, by taking advantage of more …

Decentralized optimization are playing an important role in applications such as training
large machine learning models, among others. Despite its superior practical performance …

The interest in federated learning has surged in recent research due to its unique ability to
train a global model using privacy-secured information held locally on each client. This …

We study a class of distributed optimization algorithms that aim to alleviate high
communication costs by allowing the clients to perform multiple local gradient-type training …

Modern advancements in large-scale machine learning would be impossible without the
paradigm of data-parallel distributed computing. Since distributed computing with large …