Explanation in machine learning and related fields such as artificial intelligence aims at
making machine learning models and their decisions understandable to humans. Existing …

Crowdsourcing has become a standard methodology to collect manually annotated data
such as relevance judgments at scale. On crowdsourcing platforms like Amazon MTurk or …

Crowdsourcing is being increasingly adopted as a platform to run studies with human
subjects. Running a crowdsourcing experiment involves several choices and strategies to …

When asked, large language models~(LLMs) like ChatGPT claim that they can assist with
relevance judgments but it is not clear whether automated judgments can reliably be used in …

Crowdsourcing has become a standard methodology to collect manually annotated data
such as relevance judgments at scale. On crowdsourcing platforms like Amazon MTurk or …

Crowdsourcing is a popular technique to collect large amounts of human-generated labels,
such as relevance judgments used to create information retrieval (IR) evaluation collections …

In order to evaluate the effectiveness of Information Retrieval (IR) systems it is key to collect
relevance judgments from human assessors. Crowdsourcing has successfully been used as …

Prior work in IR has found that searchers under time constraints may adapt their search
processes and perceive their task or their performance differently. In many of these prior …

In Information Retrieval evaluation, the classical approach of adopting binary relevance
judgments has been replaced by multi-level relevance judgments and by gain-based …

Test collection based evaluation represents the standard of evalua-tion for information
retrieval systems. Legal IR, more speci cally case law retrieval, has no such standard test …