Inspired by the human cognitive system, attention is a mechanism that imitates the human
cognitive awareness about specific information, amplifying critical details to focus more on …

We study the dynamics of gradient flow for training a multi-head softmax attention model for
in-context learning of multi-task linear regression. We establish the global convergence of …

This paper presents an empirical study to efficiently build named entity recognition (NER)
systems when a small amount of in-domain labeled data is available. Based upon recent …

Code summaries help developers comprehend programs and reduce their time to infer the
program functionalities during software maintenance. Recent efforts resort to deep learning …

This paper presents a comprehensive study to efficiently build named entity recognition
(NER) systems when a small number of in-domain labeled data is available. Based upon …

Vision transformers using self-attention or its proposed alternatives have demonstrated
promising results in many image related tasks. However, the underpinning inductive bias of …

The attention mechanism comes as a new entry point for improving the performance of
medical image segmentation. How to reasonably assign weights is a key element of the …

Aligned Large Language Models (LLMs) showcase remarkable versatility, capable of
handling diverse real-world tasks. Meanwhile, aligned LLMs are also expected to exhibit …

We present a theoretical analysis of the performance of transformer with softmax attention in
in-context learning with linear regression tasks. While the existing literature predominantly …

We investigate the problem of determining the predictive confidence (or, conversely,
uncertainty) of a neural classifier through the lens of low-resource languages. By training …