Since deep neural networks were developed, they have made huge contributions to
everyday lives. Machine learning provides more rational advice than humans are capable of …

We propose a general yet simple theorem describing the convergence of SGD under the
arbitrary sampling paradigm. Our theorem describes the convergence of an infinite array of …

SGD with momentum (SGDM) has been widely applied in many machine learning tasks, and
it is often applied with dynamic stepsizes and momentum weights tuned in a stagewise …

Natural gradient descent is an optimization method traditionally motivated from the
perspective of information geometry, and works well for many applications as an alternative …

We propose a stochastic variant of the classical Polyak step-size (Polyak, 1987) commonly
used in the subgradient method. Although computing the Polyak step-size requires …

Decentralized SGD is an emerging training method for deep learning known for its much
less (thus faster) communication per iteration, which relaxes the averaging step in parallel …

Recent works have shown that stochastic gradient descent (SGD) achieves the fast
convergence rates of full-batch gradient descent for over-parameterized models satisfying …

The goal of regression and classification methods in supervised learning is to minimize the
empirical risk, that is, the expectation of some loss function quantifying the prediction error …

The quick diagnosis of the novel coronavirus (COVID-19) disease is vital to prevent its
propagation and improve therapeutic outcomes. Computed tomography (CT) is believed to …

Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep
learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely …