This survey paper examines the recent advancements in AI agent implementations, with a
focus on their ability to achieve complex goals that require enhanced reasoning, planning …

Rigorous and reproducible evaluation is critical for assessing the state of the art and for
guiding scientific advances in Artificial Intelligence. Evaluation is challenging in practice due …

the researchers studying them. The LLMs may also modify our prompts and their outputs.
While this practice may serve as a guardrail against misuse, it can also have undesirable …

This study compares state-of-the-art Large Language Models (LLMs) on their tendency to
generate vulnerabilities when writing C programs using a neutral zero-shot prompt. Tihanyi …

The advancement of large language models (LLMs) relies on evaluation using public
benchmarks, but data contamination can lead to overestimated performance. Previous …

With recent advances in Large Language Models (LLMs), Agentic AI has become
phenomenal in real-world applications, moving toward multiple LLM-based agents to …

In a previous paper we defined testFAILS, a set of benchmarks for measuring the efficacy of
Large Language Models in various domains. This paper defines a second-generation …

As machine intelligence evolves, the need to test and compare the problem-solving abilities
of different AI models grows. However, current benchmarks are often simplistic, allowing …

The use of large language models (LLMs) is widespread across many domains, including
Software Engineering, where they have been used to automate tasks such as program …