Abstract Learning Rate Warmup is a popular heuristic for training neural networks,
especially at larger batch sizes, despite limited understanding of its benefits. Warmup …

Momentum has become a crucial component in deep learning optimizers, necessitating a
comprehensive understanding of when and why it accelerates stochastic gradient descent …

Adafactor is a memory efficient algorithm which does not maintain momentum and has near
0 memory overhead as compared to gradient descent. However it performs worse than …

Accelerated stochastic gradient descent (ASGD) is a workhorse in deep learning and often
achieves better generalization performance than SGD. However, existing optimization …

In distributed deep learning with data parallelism, synchronizing gradients at each training
step can cause a huge communication overhead, especially when many nodes work …

Heavy-ball momentum with decaying learning rates is widely used with SGD for optimizing
deep learning models. In contrast to its empirical popularity, the understanding of its …

Momentum-based optimizers are widely adopted for training neural networks. However, the
optimal selection of momentum coefficients remains elusive. This uncertainty impedes a …

Learning Rate Warmup is a popular heuristic for training neural networks, which downscales
early updates relative to later ones. This aids training, suggesting that the initial updates are …

Stochastic heavy ball momentum (SHB) is commonly used to train machine learning models,
and often provides empirical improvements over stochastic gradient descent. By primarily …

Understanding the dynamics of algorithms is crucial for characterizing the behavior of
trained models in modern statistical learning. This thesis presents a few recent results on …