Motivated by large-scale optimization problems arising in the context of machine learning,
there have been several advances in the study of asynchronous parallel and distributed …

Online matrix vector multiplication is a fundamental step and bottleneck in many machine
learning algorithms. It is defined as follows: given a matrix at the pre-processing phase, at …

Due to the high communication cost in distributed and federated learning problems,
methods relying on compression of communicated messages are becoming increasingly …

We present a unified framework for analyzing local SGD methods in the convex and strongly
convex regimes for distributed/federated training of supervised machine learning models …

In this paper we introduce a unified analysis of a large family of variants of proximal
stochastic gradient descent (SGD) which so far have required different intuitions …

Using backpropagation to compute gradients of objective functions for optimization has
remained a mainstay of machine learning. Backpropagation, or reverse-mode differentiation …

In this paper, we propose a unified analysis of variants of distributed SGD with arbitrary
compressions and delayed updates. Our framework is general enough to cover different …

Abstract Stochastic Gradient Descent-Ascent (SGDA) is one of the most prominent
algorithms for solving min-max optimization and variational inequalities problems (VIP) …

Deep learning has been widely used in many fields, but the model training process usually
consumes massive computational resources and time. Therefore, designing an efficient …

We introduce a new approach to develop stochastic optimization algorithms for a class of
stochastic composite and possibly nonconvex optimization problems. The main idea is to …