Overparameterized neural networks generalize well but are expensive to train. Ideally, one
would like to reduce their computational cost while retaining their generalization benefits …

We show that taking the width and depth to infinity in a deep neural network with skip
connections, when branches are scaled by $1/\sqrt {depth} $, result in the same covariance …

In this paper, we study the infinite-depth limit of finite-width residual neural networks with
random Gaussian weights. With proper scaling, we show that by fixing the width and taking …

When the parameters are independently and identically distributed (initialized) neural
networks exhibit undesirable properties that emerge as the number of layers increases, eg a …

Expressiveness and generalization of deep models was recently addressed via the
connection between neural networks (NNs) and kernel learning, where first-order dynamics …

This paper is the second in the series Commutative Scaling of Width and Depth (WD) about
commutativity of infinite width and depth limits in deep neural networks. Our aim is to …

Modern neural networks featuring a large number of layers (depth) and units per layer
(width) have achieved a remarkable performance across many domains. While there exists …

In the recent years, Deep Neural Networks (DNNs) have managed to succeed at tasks that
previously appeared impossible, such as human-level object recognition, text synthesis …

When neural network's parameters are initialized as iid, neural networks exhibit undesirable
forward and backward properties as the number of layers increases, eg, vanishing …

The recently developed link between strongly overparametrized neural networks (NNs) and
kernel methods has opened a new way to understand puzzling features of NNs, such as …