Attention mechanism is a central component of the transformer architecture which led to the
phenomenal success of large language models. However, the theoretical principles …

Since its inception in" Attention Is All You Need", transformer architecture has led to
revolutionary advancements in NLP. The attention layer within the transformer admits a …

Abstract Language modeling on large-scale datasets improves performance of various
downstream tasks. The validation pre-training loss is often used as the evaluation metric for …

In this paper we fully describe the trajectory of gradient flow over $2 $-layer diagonal linear
networks for the regression setting in the limit of vanishing initialisation. We show that the …

Recent work has highlighted the role of initialization scale in determining the structure of the
solutions that gradient methods converge to. In particular, it was shown that large …

A precise high-dimensional asymptotic theory for boosting and minimum-l1-norm
interpolated classifiers Page 1 The Annals of Statistics 2022, Vol. 50, No. 3, 1669–1695 …

As part of the effort to understand implicit bias of gradient descent in overparametrized
models, several results have shown how the training trajectory on the overparametrized …

We examine the continuous-time counterpart of mirror descent, namely mirror flow, on
classification problems which are linearly separable. Such problems are minimised 'at …

Most of the recent successful applications of neural networks have been based on training
with gradient descent updates. However, for some small networks, other mirror descent …

We study the convergence rate of Bregman gradient methods for convex optimization in the
space of measures on a d-dimensional manifold. Under basic regularity assumptions, we …