Ever since the Turing Test was proposed in the 1950s, humans have explored the mastering
of language intelligence by machine. Language is essentially a complex, intricate system of …

Metacognitive knowledge refers to humans' intuitive knowledge of their own thinking and
reasoning processes. Today's best LLMs clearly possess some reasoning processes. The …

Data curation is an essential component of large-scale pretraining. In this work, we
demonstrate that jointly selecting batches of data is more effective for learning than selecting …

This is the interim publication of the first International Scientific Report on the Safety of
Advanced AI. The report synthesises the scientific understanding of general-purpose AI--AI …

Pretraining data selection has the potential to improve language model pretraining efficiency
by utilizing higher-quality data from massive web data corpora. Current data selection …

As artificial intelligence (AI) continues to advance, it demonstrates capabilities comparable
to human intelligence, with significant potential to transform education and workforce …

As large language models (LLMs) are applied to more use cases, creating high quality, task-
specific datasets for fine-tuning becomes a bottleneck for model improvement. Using high …

Fine-tuning transformer-based deep-learning models are currently at the forefront of natural
language processing (NLP) and information retrieval (IR) tasks. However, fine-tuning these …

Large Language Model (LLM) pre-training exhausts an ever growing compute budget, yet
recent research has demonstrated that careful document selection enables comparable …

This position paper proposes a data-centric viewpoint of AI research, focusing on large
language models (LLMs). We start by making the key observation that data is instrumental in …