Graph neural networks (GNNs) are an emerging research field. This specialized deep
neural network architecture is capable of processing graph structured data and bridges the …

Stochastic optimization lies at the heart of machine learning, and its cornerstone is
stochastic gradient descent (SGD), a method introduced over 60 years ago. The last eight …

Machine learning develops rapidly, which has made many theoretical breakthroughs and is
widely applied in various fields. Optimization, as an important part of machine learning, has …

Abstract Machine learning models are not static and may need to be retrained on slightly
changed datasets, for instance, with the addition or deletion of a set of data points. This has …

Today, a largely scalable computing environment provides a possibility of carrying out
various data-intensive natural language processing and machine-learning tasks. One of …

Incremental learning refers to learning from streaming data, which arrive over time, with
limited memory resources and, ideally, without sacrificing model accuracy. This setting fits …

The question of how to incorporate curvature information into stochastic approximation
methods is challenging. The direct application of classical quasi-Newton updating …

We propose a new stochastic L-BFGS algorithm and prove a linear convergence rate for
strongly convex and smooth functions. Our algorithm draws heavily from a recent stochastic …

The standard L-BFGS method relies on gradient approximations that are not dominated by
noise, so that search directions are descent directions, the line search is reliable, and quasi …

In this paper we study stochastic quasi-Newton methods for nonconvex stochastic
optimization, where we assume that noisy information about the gradients of the objective …