Abstract Model complexity is a fundamental problem in deep learning. In this paper, we
conduct a systematic overview of the latest studies on model complexity in deep learning …

This paper surveys the machine learning literature and presents in an optimization
framework several commonly used machine learning approaches. Particularly …

Neural Architecture Search (NAS) has been explosively studied to automate the discovery of
top-performer neural networks. Current works require heavy training of supernet or intensive …

We introduce a new class of time-continuous recurrent neural network models. Instead of
declaring a learning system's dynamics by implicit nonlinearities, we construct networks of …

In this paper we investigate the family of functions representable by deep neural networks
(DNN) with rectified linear units (ReLU). We give an algorithm to train a ReLU DNN with one …

We present strong mixed-integer programming (MIP) formulations for high-dimensional
piecewise linear functions that correspond to trained neural networks. These formulations …

The success of deep networks has been attributed in part to their expressivity: per
parameter, deep networks can approximate a richer class of functions than shallow …

Abstract Deep Neural Networks (DNNs) are very popular these days, and are the subject of
a very intense investigation. A DNN is made up of layers of internal units (or neurons), each …

We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized
fully connected network N with ReLU activations. Our results show that the empirical …

Accuracy predictor is a key component in Neural Architecture Search (NAS) for ranking
architectures. Building a high-quality accuracy predictor usually costs enormous …