Transformers have achieved great success in many artificial intelligence fields, such as
natural language processing, computer vision, and audio processing. Therefore, it is natural …

With the rapid advancement of information technology, online information has been
exponentially growing day by day, especially in the form of text documents such as news …

Transformers-based models, such as BERT, have been one of the most successful deep
learning models for NLP. Unfortunately, one of their core limitations is the quadratic …

Transformer-based models are unable to process long sequences due to their self-attention
operation, which scales quadratically with the sequence length. To address this limitation …

Biological systems understand the world by simultaneously processing high-dimensional
inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The …

Transformer models have advanced the state of the art in many Natural Language
Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended …

Artificial intelligence (AI) has been applied widely in our daily lives in a variety of ways with
numerous success stories. AI has also contributed to dealing with the coronavirus disease …

Transformers have achieved success in both language and vision domains. However, it is
prohibitively expensive to scale them to long sequences such as long documents or high …

Symbolic music generation aims to generate music scores automatically. A recent trend is to
use Transformer or its variants in music generation, which is, however, suboptimal, because …

Multivariate time series forecasting focuses on predicting future values based on historical
context. State-of-the-art sequence-to-sequence models rely on neural attention between …