Neural sequence models based on the transformer architecture have demonstrated
remarkable\emph {in-context learning}(ICL) abilities, where they can perform new tasks …

We study the dynamics of gradient flow for training a multi-head softmax attention model for
in-context learning of multi-task linear regression. We establish the global convergence of …

Large language models (LLMs) demonstrate impressive reasoning abilities, but translating
reasoning into actions in the real world remains challenging. In particular, it remains unclear …

Despite the great success of Transformer networks in various applications such as natural
language processing and computer vision, their theoretical aspects are not well understood …

We study the phenomenon of in-context learning (ICL) exhibited by large language models,
where they can adapt to a new learning task, given a handful of labeled examples, without …

In-context learning (ICL) is a cornerstone of large language model (LLM) functionality, yet its
theoretical foundations remain elusive due to the complexity of transformer architectures. In …

When training deep neural networks, a model's generalization error is often observed to
follow a power scaling law dependent both on the model size and the data size. Perhaps the …

This paper provides norm-based generalization bounds for the Transformer architecture that
do not depend on the input sequence length. We employ a covering number based …

Large language models (LLMs) demonstrate impressive reasoning abilities, but translating
reasoning into actions in the real world remains challenging. In particular, it is unclear how …

Actor-critic (AC) algorithms have been the powerhouse behind many successful yet
challenging applications. However, the theoretical understanding of finite-time convergence …