Contemporary evaluation techniques are inadequate for agentic systems. These
approaches either focus exclusively on final outcomes--ignoring the step-by-step nature of …

The rapid advancement of Large Language Models (LLMs) has driven their expanding
application across various fields. One of the most promising applications is their role as …

Reward modeling is crucial for aligning large language models (LLMs) with human
preferences, especially in reinforcement learning from human feedback (RLHF). However …

Experimental evaluations of software engineering innovations, eg, tools and processes,
often include human-subject studies as a component of a multi-pronged strategy to obtain …

Objective: The paper introduces a framework for the evaluation of the encoding of factual
scientific knowledge, designed to streamline the manual evaluation process typically …

The rapid development of Large Language Models (LLMs) for healthcare applications has
spurred calls for holistic evaluation beyond frequently-cited benchmarks like USMLE, to …

Personalization of Large Language Models (LLMs) has recently become increasingly
important with a wide range of applications. Despite the importance and recent progress …

High quality annotations are increasingly a bottleneck in the explosively growing machine
learning ecosystem. Scalable evaluation methods that avoid costly annotation have …

This whitepaper offers an overview of the ethical considerations surrounding research into
or with large language models (LLMs). As LLMs become more integrated into widely used …

Large Language Models (LLMs) have demonstrated remarkable capabilities in performing
tasks across various domains without needing explicit retraining. This capability, known as …