We present PYHESSIAN, a new scalable framework that enables fast computation of
Hessian (ie, second-order derivative) information for deep neural networks. PYHESSIAN …

First-order stochastic methods are the state-of-the-art in large-scale machine learning
optimization owing to efficient per-iteration complexity. Second-order methods, while able to …

We consider variants of trust-region and adaptive cubic regularization methods for non-
convex optimization, in which the Hessian matrix is approximated. Under certain condition …

Preconditioned gradient methods are among the most general and powerful tools in
optimization. However, preconditioning requires storing and manipulating prohibitively large …

The paper studies the solution of stochastic optimization problems in which approximations
to the gradient and Hessian are obtained through subsampling. We first consider Newton …

We propose a novel limited-memory stochastic block BFGS update for incorporating
enriched curvature information in stochastic approximation methods. In our method, the …

Differentially private (stochastic) gradient descent is the workhorse of DP private machine
learning in both the convex and non-convex settings. Without privacy constraints, second …

Numerous intriguing optimization problems arise as a result of the advancement of machine
learning. The stochastic first-order method is the predominant choice for those problems due …

We consider the minimization of non-convex functions that typically arise in machine
learning. Specifically, we focus our attention on a variant of trust region methods known as …

For large-scale finite-sum minimization problems, we study non-asymptotic and high-
probability global as well as local convergence properties of variants of Newton's method …