While Transformers have been the main architecture behind deep learning's success in
language modeling, state-space models (SSMs) such as Mamba have recently been shown …

Transformers with linear attention allow for efficient parallel training but can simultaneously
be formulated as an RNN with 2D (matrix-valued) hidden states, thus enjoying linear-time …

Linear RNN architectures, like Mamba, can be competitive with Transformer models in
language modeling while having advantageous deployment characteristics. Given the focus …

A wide array of sequence models are built on a framework modeled after Transformers,
comprising alternating sequence mixer and channel mixer layers. This paper studies a …

Efficiently modeling sequences with infinite context length has been a long-standing
problem. Past works suffer from either the quadratic computation complexity or the limited …

Various linear complexity models, such as Linear Transformer (LinFormer), State Space
Model (SSM), and Linear RNN (LinRNN), have been proposed to replace the conventional …

We study continued training and supervised fine-tuning (SFT) of a language model (LM) to
make effective use of long-context information. We first establish a reliable evaluation …

Transformer, a deep neural network architecture, has long dominated the field of natural
language processing and beyond. Nevertheless, the recent introduction of Mamba …

In the rapidly evolving field of deep learning, the demand for models that are both
expressive and computationally efficient has never been more critical. This paper introduces …

Recurrent large language models that compete with Transformers in language modeling
perplexity are emerging at a rapid rate (eg, Mamba, RWKV). Excitingly, these architectures …