The recently proposed stochastic Polyak stepsize (SPS) and stochastic line-search (SLS) for
SGD have shown remarkable effectiveness when training over-parameterized models …

We consider the problem of estimating the learning rate in adaptive methods, such as
AdaGrad and Adam. We propose Prodigy, an algorithm that provably estimates the distance …

This paper proposes a new easy-to-implement parameter-free gradient-based optimizer:
DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient---matching the …

Adaptive algorithms like AdaGrad and AMSGrad are successful in nonconvex optimization
owing to their parameter-agnostic ability–requiring no a priori knowledge about problem …

Tuning hyperparameters, such as the stepsize, presents a major challenge of training
machine learning models. To address this challenge, numerous adaptive optimization …

State-of-the-art federated learning algorithms such as FedAvg require carefully tuned
stepsizes to achieve their best performance. The improvements proposed by existing …

Abstract Stochastic Gradient Descent (SGD) is one of the many iterative optimization
methods that are widely used in solving machine learning problems. These methods display …

Training a modern machine learning architecture on a new task requires extensive learning-
rate tuning, which comes at a high computational cost. Here we develop new adaptive …

Adaptive optimization methods are widely recognized as among the most popular
approaches for training Deep Neural Networks (DNNs). Techniques such as Adam …

Optimization methods play a crucial role in modern machine learning, powering the
remarkable empirical achievements of deep learning models. These successes are even …