With the emergence of various information access systems exhibiting increasing complexity,
there is a critical need for sound and scalable means of automatic evaluation. To address …

Dealing with unjudged documents (" holes") in relevance assessments is a perennial
problem when evaluating search systems with offline experiments. Holes can reduce the …

The purpose of the Strategic Workshop in Information Retrieval in Lorne is to explore the
long-range issues of the Information Retrieval field, to recognize challenges that are on-or …

Traditionally, machine learning algorithms relied on reliable labels from experts to build
predictions. More recently however, algorithms have been receiving data from the general …

This paper explores the utility of a Large Language Model (LLM) to automatically generate
queries and query variants from a description of an information need. Given a set of …

We undertake a detailed examination of the steps that make up offline experiments for
recommender system evaluation, including the manner in which the available ratings are …

Offline evaluation is an essential complement to online experiments in the selection,
improvement, tuning, and deployment of recommender systems. Offline methodologies for …

Web Search Engine Result Pages (SERPs) are complex responses to queries, containing
many heterogeneous result elements (web results, advertisements, and specialised" …

Recent advances in large language models have enabled the development of viable
generative retrieval systems. Instead of a traditional document ranking, generative retrieval …

We present ir-measures, a new tool that makes it convenient to calculate a diverse set of
evaluation measures used in information retrieval. Rather than implementing its own …