This open-source book represents our attempt to make deep learning approachable,
teaching readers the concepts, the context, and the code. The entire book is drafted in …

Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a
Markov chain with a stationary distribution. With this perspective, we derive several new …

Accelerated gradient methods play a central role in optimization, achieving optimal rates in
many settings. Although many generalizations and extensions of Nesterov's original …

Gradient-based optimization algorithms can be studied from the perspective of limiting
ordinary differential equations (ODEs). Motivated by the fact that existing ODEs do not …

Why Momentum Really Works Distill About Prize Submit Why Momentum Really Works Step-size
α = 0.02 Momentum β = 0.99 We often think of Momentum as a means of dampening …

The success of the Adam optimizer on a wide array of architectures has made it the default
in settings where stochastic gradient descent (SGD) performs poorly. However, our …

We consider the random-design least-squares regression problem within the reproducing
kernel Hilbert space (RKHS) framework. Given a stream of independent and identically …

We consider the optimization of a quadratic objective function whose gradients are only
accessible through a stochastic oracle that returns the gradient at any given point plus a …

Abstract Stochastic Gradient Descent (SGD) is an important algorithm in machine learning.
With constant learning rates, it is a stochastic process that, after an initial phase of …

In this paper, we adapt the control theoretic concept of dissipativity theory to provide a
natural understanding of Nesterov's accelerated method. Our theory ties rigorous …