Despite the remarkable empirical performance of Transformers, their theoretical
understanding remains elusive. Here, we consider a deep multi-head self-attention network …

In-context learning (ICL) of large language models has proven to be a surprisingly effective
method of learning a new task from only a few demonstrative examples. In this paper, we …

In-context learning (ICL) refers to a remarkable capability of pretrained large language
models, which can learn a new task given a few examples during inference. However …

Transformers have demonstrated great power in the recent development of large
foundational models. In particular, the Vision Transformer (ViT) has brought revolutionary …

In-context learning has been recognized as a key factor in the success of Large Language
Models (LLMs). It refers to the model's ability to learn patterns on the fly from provided in …

Large language models based on the Transformer architecture have demonstrated
impressive capabilities to learn in context. However, existing theoretical studies on how this …

Transformers can efficiently learn in-context from example demonstrations. Most existing
theoretical analyses studied the in-context learning (ICL) ability of transformers for linear …

Group imbalance has been a known problem in empirical risk minimization (ERM), where
the achieved high average accuracy is accompanied by low accuracy in a minority group …

Autoregressively trained transformers have brought a profound revolution to the world,
especially with their in-context learning (ICL) ability to address downstream tasks. Recently …