The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of
information (the" needle") from long distractor texts (the" haystack"), has been widely …

Long-context modeling capabilities of Large Language Models (LLMs) have garnered
widespread attention, leading to the emergence of LLMs with ultra-context windows …

As large language models (LLMs) advance, it becomes more challenging to reliably
evaluate their output due to the high costs of human evaluation. To make progress towards …

We study continued training and supervised fine-tuning (SFT) of a language model (LM) to
make effective use of long-context information. We first establish a reliable evaluation …

Extending context window sizes allows large language models (LLMs) to process longer
sequences and handle more complex tasks. Rotary Positional Embedding (RoPE) has …

Recently, Large language models (LLMs) have revolutionized Natural Language
Processing (NLP). Pretrained LLMs, due to limited training context size, struggle with …

Large language models (LLMs) have achieved substantial progress in processing long
contexts but still struggle with long-context reasoning. Existing approaches typically involve …

Existing benchmarks for evaluating long-context language models (LCLMs) primarily focus
on long-context recall, requiring models to produce short responses based on a few critical …

We introduce FACTS Grounding, an online leaderboard and associated benchmark that
evaluates language models' ability to generate text that is factually accurate with respect to …

Automatically summarizing large text collections is a valuable tool for document research,
with applications in journalism, academic research, legal work, and many other fields. In this …