Recently, the impressive empirical success of policy gradient (PG) methods has catalyzed
the development of their theoretical foundations. Despite the huge efforts directed at the …

In this paper we study several classes of stochastic optimization algorithms enriched with
heavy ball momentum. Among the methods studied are: stochastic gradient descent …

We provide an improved analysis of normalized SGD showing that adding momentum
provably removes the need for large batch sizes on non-convex objectives. Then, we …

We consider non-convex stochastic optimization using first-order algorithms for which the
gradient estimates may have heavy tails. We show that a combination of gradient clip** …

Momentum is known to accelerate the convergence of gradient descent in strongly convex
settings without stochastic gradient noise. In stochastic optimization, such as training neural …

Abstract Sign Stochastic Gradient Descent (signSGD) is a communication-efficient stochastic
algorithm that only uses the sign information of the stochastic gradient to update the model's …

Recognizing new objects by learning from a few labeled examples in an evolving
environment is crucial to obtain excellent generalization ability for real-world machine …

Online optimization has witnessed a massive surge of research attention in recent years. In
this paper, we propose online gradient descent and online bandit algorithms over …

In this paper, we consider Riemannian online convex optimization with dynamic regret. First,
we propose two novel algorithms, namely the Riemannian Online Optimistic Gradient …

We study the problem of solving strongly convex and smooth unconstrained optimization
problems using stochastic first-order algorithms. We devise a novel algorithm, referred to …