How can modern neural networks like language models be useful to the field of language
acquisition, and more broadly cognitive science, if they are not a priori designed to be …

LLM-as-a-Judge presents a promising alternative to human evaluators across various tasks,
but inherent biases, especially position bias-a tendency to favor solutions based on their …

Large Language Models (LLMs) excel in stand-alone code tasks like HumanEval and
MBPP, but struggle with handling entire code repositories. This challenge has prompted …

The LLM Agent, equipped with a code interpreter, is capable of automatically solving real-
world coding tasks, such as data analysis and image editing. However, existing benchmarks …

We interact with computers on an everyday basis, be it in everyday life or work, and many
aspects of work can be done entirely with access to a computer and the Internet. At the same …

Large Language Models (LLMs) show promise in code generation tasks. However, their
code-writing abilities are often limited in scope: while they can successfully implement …

In recent years, researchers have proposed numerous benchmarks to evaluate the
impressive coding capabilities of large language models (LLMs). However, current …

With the goal of benchmarking generative systems beyond expert software development
ability, we introduce Commit0, a benchmark that challenges AI agents to write libraries from …

Testing plays a crucial role in the software development cycle, enabling the detection of
bugs, vulnerabilities, and other undesirable behaviors. To perform software testing, testers …

Large Language Models have advanced automated software development, however, it
remains a challenge to correctly infer dependencies, namely, identifying the internal …