The purpose of this article is to review the achievements made in the last few years towards
the understanding of the reasons behind the success and subtleties of neural network …

We study the implicit bias of gradient flow (ie, gradient descent with infinitesimal step size)
on linear neural network training. We propose a tensor formulation of neural networks that …

Training deep neural networks with stochastic gradient descent (SGD) can often achieve
zero training loss on real-world tasks although the optimization landscape is known to be …

Existing analyses of optimization in deep learning are either continuous, focusing on
(variants of) gradient flow, or discrete, directly treating (variants of) gradient descent …

Residual neural networks are state-of-the-art deep learning models. Their continuous-depth
analog, neural ordinary differential equations (ODEs), are also widely used. Despite their …

Neural networks with wide layers have attracted significant attention due to their
equivalence to Gaussian processes, enabling perfect fitting of training data while …

We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD)
for training $ L $-hidden-layer linear residual networks (ResNets). We prove that for training …

In this paper, we analyze the convergence of gradient flow on a multi-layer linear model with
a loss function of the form $ f (W_1W_2\cdots W_L) $. We show that when $ f $ satisfies the …

Incorporating a so-called “momentum” dynamic in gradient descent methods is widely used
in neural net training as it has been broadly observed that, at least empirically, it often leads …

We study the convergence properties of gradient descent for training deep linear neural
networks, ie, deep matrix factorizations, by extending a previous analysis for the related …