Optimization is a critical component in deep learning. We think optimization for neural
networks is an interesting topic for theoretical research due to various reasons. First, its …

This survey describes probabilistic algorithms for linear algebraic computations, such as
factorizing matrices and solving linear systems. It focuses on techniques that have a proven …

This paper provides a review and commentary on the past, present, and future of numerical
optimization algorithms in the context of machine learning applications. Through case …

Coordinate descent algorithms solve optimization problems by successively performing
approximate minimization along coordinate directions or coordinate hyperplanes. They have …

When and why can a neural network be successfully trained? This article provides an
overview of optimization algorithms and theory for training neural networks. First, we discuss …

Since its beginning, optimization has played a vital role in data science. The analysis and
solution methods for many statistical and machine learning models rely on optimization. The …

The standard assumption for proving linear convergence of first order methods for smooth
convex optimization is the strong convexity of the objective function, an assumption which …

In this paper we propose new methods for solving huge-scale optimization problems. For
problems of this size, even the simplest full-dimensional vector operations are very …

In this paper we develop a randomized block-coordinate descent method for minimizing the
sum of a smooth and a simple nonsmooth block-separable convex function and prove that it …

We develop a novel, fundamental, and surprisingly simple randomized iterative method for
solving consistent linear systems. Our method has six different but equivalent interpretations …