Rigorous and reproducible evaluation is critical for assessing the state of the art and for
guiding scientific advances in Artificial Intelligence. Evaluation is challenging in practice due …

We introduce Holmes, a new benchmark designed to assess language models'(LMs')
linguistic competence—their unconscious understanding of linguistic phenomena …

Evaluating the reasoning abilities of large language models (LLMs) is challenging. Existing
benchmarks often depend on static datasets, which are vulnerable to data contamination …