The asynchronous parallel implementations of stochastic gradient (SG) have been broadly
used in solving deep neural network and received many successes in practice recently …

Stochastic gradient descent (SGD) is a ubiquitous algorithm for a variety of machine learning
problems. Researchers and industry have developed several techniques to optimize SGD's …

Distributed optimization methods for large-scale machine learning suffer from a
communication bottleneck. It is difficult to reduce this bottleneck while still efficiently and …

We develop a new family of variance reduced stochastic gradient descent methods for
minimizing the average of a very large number of smooth functions. Our method—JacSketch …

This monograph presents a class of algorithms called coordinate descent algorithms for
mathematicians, statisticians, and engineers outside the field of optimization. This particular …

Asynchronous parallel optimization received substantial successes and extensive attention
recently. One of core theoretical questions is how much speedup (or benefit) the …

This paper presents a Parallel Block-Coordinate Descent (PBCD) algorithm for solving the
user equilibrium traffic assignment problem. Most of the existing algorithms for the user …

One of the most widely used optimization methods for large-scale machine learning
problems is distributed asynchronous stochastic gradient descent (DASGD). However, a key …

Multi-task learning aims to learn multiple tasks jointly by exploiting their relatedness to
improve the generalization performance for each task. Traditionally, to perform multi-task …

Relative smoothness—a notion introduced in Birnbaum et al.(Proceedings of the 12th ACM
conference on electronic commerce, ACM, pp 127–136, 2011) and recently rediscovered in …