Optimization is a critical component in deep learning. We think optimization for neural
networks is an interesting topic for theoretical research due to various reasons. First, its …

Diffusion models have proven to be highly effective in generating high-quality images.
However, adapting large pre-trained diffusion models to new domains remains an open …

When and why can a neural network be successfully trained? This article provides an
overview of optimization algorithms and theory for training neural networks. First, we discuss …

A recent line of research has shown that gradient-based algorithms with random
initialization can converge to the global minima of the training loss for over-parameterized …

Natural gradient descent has proven very effective at mitigating the catastrophic effects of
pathological curvature in the objective function, but little is known theoretically about its …

We study the problem of learning one-hidden-layer neural networks with Rectified Linear
Unit (ReLU) activation function, where the inputs are sampled from standard Gaussian …

Empirical studies show that gradient-based methods can learn deep neural networks
(DNNs) with very good generalization performance in the over-parameterization regime …

As science and engineering have become increasingly data-driven, the role of optimization
has expanded to touch almost every stage of the data analysis pipeline, from signal and …

The multi-head attention layer is one of the key components of the transformer architecture
that sets it apart from traditional feed-forward models. Given a sequence length $ k …

We consider the problem of learning an unknown ReLU network with respect to Gaussian
inputs and obtain the first nontrivial results for networks of depth more than two. We give an …