Neural networks have shown tremendous growth in recent years to solve numerous
problems. Various types of neural networks have been introduced to deal with different types …

The activation function plays an important role in the training and the performance of an
Artificial Neural Network. They provide the necessary non-linear properties to any Artificial …

While contrastive approaches of self-supervised learning (SSL) learn representations by
minimizing the distance between two augmented views of the same data point (positive …

This open-source book represents our attempt to make deep learning approachable,
teaching readers the concepts, the context, and the code. The entire book is drafted in …

We perform a careful, thorough, and large scale empirical study of the correspondence
between wide neural networks and kernel methods. By doing so, we resolve a variety of …

Deep networks often suffer from vanishing or exploding gradients due to inefficient signal
propagation, leading to long training times or convergence difficulties. Various architecture …

Transformers have proved effective in many NLP tasks. However, their training requires non-
trivial efforts regarding designing cutting-edge optimizers and learning rate schedulers …

Normalization layers are a staple in state-of-the-art deep neural network architectures. They
are widely believed to stabilize training, enable higher learning rate, accelerate …

Training neural networks under a strict Lipschitz constraint is useful for provable adversarial
robustness, generalization bounds, interpretable gradients, and Wasserstein distance …

The recent striking success of deep neural networks in machine learning raises profound
questions about the theoretical principles underlying their success. For example, what can …