Large language models (LLMs) demonstrate powerful capabilities, but they still face
challenges in practical applications, such as hallucinations, slow knowledge updates, and …

Natural Language Generation (NLG) has improved exponentially in recent years thanks to
the development of sequence-to-sequence deep learning technologies such as Transformer …

While large language models (LLMs) have demonstrated remarkable capabilities across a
range of downstream tasks, a significant concern revolves around their propensity to exhibit …

Since the proposal of transformers, these models have been limited to bounded input
lengths, because of their need to attend to every token in the input. In this work, we propose …

In this work, we propose Retentive Network (RetNet) as a foundation architecture for large
language models, simultaneously achieving training parallelism, low-cost inference, and …

Text summarization has been a crucial problem in natural language processing (NLP) for
several decades. It aims to condense lengthy documents into shorter versions while …

The quadratic complexity and weak length extrapolation of Transformers limits their ability to
scale to long sequences, and while sub-quadratic solutions like linear attention and state …

We fine-tune large language models to write natural language critiques (natural language
critical comments) using behavioral cloning. On a topic-based summarization task, critiques …

We present a series of long-context LLMs that support effective context windows of up to
32,768 tokens. Our model series are built through continual pretraining from Llama 2 with …

Extending the context window of large language models (LLMs) is getting popular recently,
while the solution of augmenting LLMs with retrieval has existed for years. The natural …