Test set contamination, wherein test data from a benchmark ends up in a newer model's
training set, is a well-documented obstacle for fair LLM evaluation and can quickly render …

We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging
mathematics problems crafted and vetted by expert mathematicians. The questions cover …

As language models regularly make mistakes when solving math problems, automated
identification of errors in the reasoning process becomes increasingly significant for their …

The rapid advancement of Large Language Models (LLMs) has demonstrated remarkable
progress in complex reasoning tasks. However, a significant discrepancy persists between …

With the increasing code reasoning capabilities of existing large language models (LLMs)
and breakthroughs in reasoning models like OpenAI o1 and o3, there is a growing need to …

Recent progress in large language models (LLM) found chain-of-thought prompting
strategies to improve the reasoning ability of LLMs by encouraging problem solving through …

Recent advancements in language models have led to significant improvements in
mathematical reasoning across various benchmarks. However, most of these benchmarks …

We present a method of generating first-order logic statements whose complexity can be
controlled along multiple dimensions. We use this method to automatically create several …

Large language models (LLMs) demonstrate exceptional performance on complex
reasoning tasks. However, despite their strong reasoning capabilities in high-resource …

Synthetic high-quality multi-step reasoning data can significantly enhance the performance
of large language models on various tasks. However, most existing methods rely on …