Quality improvement methods are essential to gathering high-quality crowdsourced data,
both for research and industry applications. A popular and broadly applicable method is task …

Human evaluation is critical for validating the performance of text-to-image generative
models, as this highly cognitive process requires deep comprehension of text and images …

Human annotated data plays a crucial role in machine learning (ML) research and
development. However, the ethical considerations around the processes and decisions that …

Crowdsourcing markets provide workers with a centralized place to find paid work. What
may not be obvious at first glance is that, in addition to the work they do for pay, crowd …

Purpose Online labour markets (OLMs) have recently become a widespread phenomenon
of digital work. While the implications of OLMs on worker well-being are hotly debated, little …

To understand human behavior, social scientists need people and data. In the last decade,
Amazon's Mechanical Turk (MTurk) emerged as a flexible, affordable, and reliable source of …

The increased use of machine learning in recent years led to large volumes of data being
manually labeled via crowdsourcing microtasks completed by humans. This brought about …

Image-Text matching (ITM) is a common task for evaluating the quality of Vision and
Language (VL) models. However, existing ITM benchmarks have a significant limitation …

Accurate task pricing in microtask marketplaces requires substantial effort via trial and error,
contributing to a pattern of worker underpayment. In response, we introduce Fair Work …

We present the results of a survey fielded in June of 2022 as a lens to examine recent data
reliability issues on Amazon Mechanical Turk. We contrast bad data from this survey with …