Crowdsourcing, a distributed human problem-solving paradigm is an active research area
which has attracted significant attention in the fields of computer science, business, and …

Abstract Many tasks in Natural Language Processing (NLP) and Computer Vision (CV) offer
evidence that humans disagree, from objective tasks such as part-of-speech tagging to more …

Despite the impressive performance of Artificial Intelligence (AI) systems, their robustness
remains elusive and constitutes a key issue that impedes large-scale adoption. Besides …

The lack of publicly available ground-truth data has been identified as the major challenge
for transferring recent developments in deep learning to the biomedical imaging domain …

Crowdsourcing provides a scalable and efficient way to construct labeled datasets for
training machine learning systems. However, creating comprehensive label guidelines for …

Data science and machine learning provide indispensable techniques for understanding
phenomena at scale, but the discretionary choices made when doing this work are often not …

In this paper, we address the concept of" alignment" in large language models (LLMs)
through the lens of post-structuralist socio-political theory, specifically examining its parallels …

Visual question answering is the task of returning the answer to a question about an image.
A challenge is that different people often provide different answers to the same visual …

Data is received characterizing a request for agent compu tation of sensor data. The request
includes a required con fidence and required latency for completion of the agent …

The process of qualitative coding often involves multiple coders coding the same data to
ensure reliable codes and a consistent understanding of the codebook. One aspect of …