Large Language Models (LLMs) applied to code-related applications have emerged as a
prominent field, attracting significant interest from both academia and industry. However, as …

We present CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation), a
benchmark consisting of 800 Python functions (3-13 lines). Each function comes with an …

Transformers have revolutionized natural language processing (NLP) and have had a huge
impact on automating tasks. Recently, transformers have led to the development of powerful …

Well-trained machine-learning models, which leverage large amounts of open-source
software data, have now become an interesting approach to automating many software …

Large Language Models (LLMs) are increasingly becoming fundamental in supporting
software developers in coding tasks. The massive datasets used for training LLMs are often …

We study the generalization abilities of language models when translating natural language
into formal specifications with complex semantics. In particular, we fine-tune language …

Code generation models have increasingly become integral to aiding software
development. Although current research has thoroughly examined the correctness of the …

While language models are increasingly more proficient at code generation, they still
frequently generate incorrect programs. Many of these programs are obviously wrong, but …

Recent advancements in large language models (LLMs) have greatly improved code
generation, specifically at the function level. For instance, GPT-4o has achieved a 91.0 …

We present The Vault, a dataset of high-quality code-text pairs in multiple programming
languages for training large language models to understand and generate code. We present …