The practical successes of deep neural networks have not been matched by theoretical
progress that satisfyingly explains their behavior. In this work, we study the information …

This paper provides a review and commentary on the past, present, and future of numerical
optimization algorithms in the context of machine learning applications. Through case …

This book builds on and is a sequel to our book Targeted Learning: Causal Inference for
Observational and Experimental Studies (2011). Since the publication of this first book on …

Natural gradient descent is an optimization method traditionally motivated from the
perspective of information geometry, and works well for many applications as an alternative …

Chapter 1 strongly advocates the stochastic back-propagation method to train neural
networks. This is in fact an instance of a more general technique called stochastic gradient …

A label consistent K-SVD (LC-KSVD) algorithm to learn a discriminative dictionary for sparse
coding is presented. In addition to using class labels of training data, we also associate label …

We improve a recent gurantee of Bach and Moulines on the linear convergence of SGD for
smooth and strongly convex objectives, reducing a quadratic dependence on the strong …

Stochastic Gradient Descent (SGD) has become popular for solving large scale supervised
machine learning optimization problems such as SVM, due to their strong theoretical …

During the last decade, the data sizes have grown faster than the speed of processors. In
this context, the capabilities of statistical machine learning methods is limited by the …

The question of how to incorporate curvature information into stochastic approximation
methods is challenging. The direct application of classical quasi-Newton updating …