Dealing with unjudged documents (" holes") in relevance assessments is a perennial
problem when evaluating search systems with offline experiments. Holes can reduce the …

This paper examines whether the Cranfield evaluation methodology is robust to gross
violations of the completeness assumption (ie, the assumption that all relevant documents …

Use of test collections and evaluation measures to assess the effectiveness of information
retrieval systems has its origins in work dating back to the early 1950s. Across the nearly 60 …

Automatically judging the quality of retrieval functions based on observable user behavior
holds promise for making retrieval evaluation faster, cheaper, and more user centered …

The most widely adopted approaches for evaluation of summary content follow some
protocol for comparing a summary with gold-standard human summaries, which are …

This book is a second edition, updated and expanded to explain the technologies that help
us find information on the web. Search engines and web navigation tools have become …

Crowdsourcing successfully strives to become a widely used means of collecting large-scale
scientific corpora. Many research fields, including Information Retrieval, rely on this novel …

In the field of information retrieval, one is often faced with the problem of computing the
correlation between two ranked lists. The most commonly used statistic that quantifies this …

We consider the problem of evaluating retrieval systems using incomplete judgment
information. Buckley and Voorhees recently demonstrated that retrieval systems can be …

Query performance prediction (QPP) aims to estimate the retrieval quality of a search system
for a query without human relevance judgments. Previous QPP methods typically return a …