User simulation for evaluating information access systems

K Balog, CX Zhai - Proceedings of the Annual International ACM SIGIR …, 2023 - dl.acm.org
With the emergence of various information access systems exhibiting increasing complexity,
there is a critical need for sound and scalable means of automatic evaluation. To address …

Evaluating generative ad hoc information retrieval

L Gienapp, H Scells, N Deckers, J Bevendorff… - Proceedings of the 47th …, 2024 - dl.acm.org
Recent advances in large language models have enabled the development of viable
generative retrieval systems. Instead of a traditional document ranking, generative retrieval …

The SimIIR 2.0 framework: user types, Markov model-based interaction simulation, and advanced query generation

S Zerhoudi, S Günther, K Plassmeier, T Borst… - Proceedings of the 31st …, 2022 - dl.acm.org
Simulated user retrieval system interactions enable studies with controlled user behavior. To
this end, the SimIIR framework offers static, rule-based methods. We present an extended …

CIRCLE: Multi-Turn Query Clarifications with Reinforcement Learning

P Erbacher, L Soulier - arxiv preprint arxiv:2311.02737, 2023 - arxiv.org
Users often have trouble formulating their information needs into words on the first try when
searching online. This can lead to frustration, as they may have to reformulate their queries …

Pre: A precision-recall-effort optimization framework for query simulation

S Labhishetty, CX Zhai - Proceedings of the 2022 ACM SIGIR …, 2022 - dl.acm.org
We study how to develop an interpretable query simulation framework that can potentially
explain the process a real user might have used to formulate a query and propose a novel …

User's knowledge and information needs in information retrieval evaluation

D El Zein, C da Costa Pereira - … of the 30th ACM Conference on User …, 2022 - dl.acm.org
The existing evaluation measures for information retrieval algorithms still lack awareness
about the user's cognitive aspects and their dynamics. They often consider an isolated query …

IRnator: A Framework for Discovering Users Needs from Sets of Suggestions

A Mustar, S Lamprier, B Piwowarski - Proceedings of the 2022 ACM …, 2022 - dl.acm.org
To tackle complex IR tasks, where users cannot precisely define their needs, interaction is
paramount. Both query-reformulation approaches and chatbots are limited for this type of …

[PDF][PDF] RULK: A Framework for Representing User Knowledge in Search-as-Learning.

A Câmara, D El Zein, C da Costa Pereira - DESIRES, 2022 - ceur-ws.org
When learning something new, users generally resort to search systems. By issuing queries
and examining search results, users acquire their knowledge from the content of result …

A synthetic search session generator for task-aware information seeking and retrieval

S Sarkar, C Shah - Proceedings of the Sixteenth ACM International …, 2023 - dl.acm.org
For users working on a complex search task, it is common to address different goals at
various stages of the task through query iterations. While addressing these goals, users go …

SimIIR 3: A Framework for the Simulation of Interactive and Conversational Information Retrieval

L Azzopardi, T Breuer, B Engelmann, C Kreutz… - Proceedings of the …, 2024 - dl.acm.org
Evaluating the interactions between users and systems presents many challenges.
Simulation offers a reliable, re-usable, and repeatable methodology to explore how different …