One of the most important properties of classical neural networks is how surprisingly
trainable they are, though their training algorithms typically rely on optimizing complicated …

Federated Learning (FL) is an emerging learning scheme that allows different distributed
clients to train deep neural networks together without data sharing. Neural networks have …

In the past decade, deep learning became the prevalent methodology for predictive
modeling thanks to the remarkable accuracy of deep neural networks in tasks such as …

The multi-head attention layer is one of the key components of the transformer architecture
that sets it apart from traditional feed-forward models. Given a sequence length $ k …

We contribute to a better understanding of the class of functions that is represented by a
neural network with ReLU activations and a given architecture. Using techniques from mixed …

A common method in training neural networks is to initialize all the weights to be
independent Gaussian vectors. We observe that by instead initializing the weights into …

We give superpolynomial statistical query (SQ) lower bounds for learning two-hidden-layer
ReLU networks with respect to Gaussian inputs in the standard (noise-free) model. No …

We consider the algorithmic problem of finding the optimal weights and biases for a two-
layer fully connected neural network to fit a given set of data points, also known as empirical …

We consider the well-studied problem of learning a linear combination of $ k $ ReLU
activations with respect to a Gaussian distribution on inputs in $ d $ dimensions. We give the …

We study the power of query access for the fundamental task of agnostic learning under the
Gaussian distribution. In the agnostic model, no assumptions are made on the labels of the …