Recommender systems exploit interaction history to estimate user preference, having been
heavily used in a wide range of industry applications. However, static recommendation …

With the rapid adoption of online shop**, academic research in the eCommerce domain
has gained traction. However, significant research challenges remain, spanning from classic …

A major obstacle to the long-term impact of most shared tasks is their lack of reproducibility.
Often only the test collections and the papers of the organizers and participants are …

We integrate irdatasets, ir_measures, and PyTerrier with TIRA in the Information Retrieval
Experiment Platform (TIREx) to promote more standardized, reproducible, scalable, and …

With the emergence of various information access systems exhibiting increasing complexity,
there is a critical need for sound and scalable means of automatic evaluation. To address …

We consider the novel problem of evaluating a recommendation policy offline in
environments where the reward signal is non-stationary. Non-stationarity appears in many …

Simulation is used as a low-cost and repeatable means of experimentation. As Information
Retrieval (IR) researchers, we are no strangers to the idea of using simulation within our …

Conversational AI is an emerging field of computer science that engages multiple research
communities, from information retrieval to natural language processing to dialogue systems …

In this paper, we discuss the benefits and challenges of shared tasks as a teaching method.
A shared task is a scientific event and a friendly competition to solve a research problem, the …

Evaluating retrieval performance without editorial relevance judgments is challenging, but
instead, user interactions can be used as relevance signals. Living labs offer a way for small …