Area under the ROC curve, aka AUC, is a measure of choice for assessing the performance
of a classifier for imbalanced data. AUC maximization refers to a learning paradigm that …

In deep learning, different kinds of deep networks typically need different optimizers, which
have to be chosen after multiple trials, making the training process inefficient. To relieve this …

In this paper, we provide a rigorous proof of convergence of the Adaptive Moment Estimate
(Adam) algorithm for a wide class of optimization objectives. Despite the popularity and …

Ever since\citet {reddi2019convergence} pointed out the divergence issue of Adam, many
new variants have been designed to obtain convergence. However, vanilla Adam remains …

Bilevel optimization has been widely applied in many important machine learning
applications such as hyperparameter optimization and meta-learning. Recently, several …

Bilevel optimization, the problem of minimizing a value function which involves the arg-
minimum of another function, appears in many areas of machine learning. In a large scale …

Standard federated optimization methods successfully apply to stochastic problems with
single-level structure. However, many contemporary ML problems-including adversarial …

Abstract Recently, Arjevani et al.[1] establish a lower bound of iteration complexity for the
first-order optimization under an $ L $-smooth condition and a bounded noise variance …

In this paper, we propose a novel Hessian inverse free Fully Single Loop Algorithm (FSLA)
for bilevel optimization problems. Classic algorithms for bilevel optimization admit a double …

We study convergence rates of AdaGrad-Norm as an exemplar of adaptive stochastic
gradient methods (SGD), where the step sizes change based on observed stochastic …