A survey on task assignment in crowdsourcing
Quality improvement methods are essential to gathering high-quality crowdsourced data,
both for research and industry applications. A popular and broadly applicable method is task …
both for research and industry applications. A popular and broadly applicable method is task …
Designing for hybrid intelligence: A taxonomy and survey of crowd-machine interaction
With the widespread availability and pervasiveness of artificial intelligence (AI) in many
application areas across the globe, the role of crowdsourcing has seen an upsurge in terms …
application areas across the globe, the role of crowdsourcing has seen an upsurge in terms …
All those wasted hours: On task abandonment in crowdsourcing
Crowdsourcing has become a standard methodology to collect manually annotated data
such as relevance judgments at scale. On crowdsourcing platforms like Amazon MTurk or …
such as relevance judgments at scale. On crowdsourcing platforms like Amazon MTurk or …
The impact of task abandonment in crowdsourcing
Crowdsourcing has become a standard methodology to collect manually annotated data
such as relevance judgments at scale. On crowdsourcing platforms like Amazon MTurk or …
such as relevance judgments at scale. On crowdsourcing platforms like Amazon MTurk or …
Crowd worker strategies in relevance judgment tasks
Crowdsourcing is a popular technique to collect large amounts of human-generated labels,
such as relevance judgments used to create information retrieval (IR) evaluation collections …
such as relevance judgments used to create information retrieval (IR) evaluation collections …
Playing planning poker in crowds: human computation of software effort estimates
M Alhamed, T Storer - 2021 IEEE/ACM 43rd International …, 2021 - ieeexplore.ieee.org
Reliable cost effective effort estimation remains a considerable challenge for software
projects. Recent work has demonstrated that the popular Planning Poker practice can …
projects. Recent work has demonstrated that the popular Planning Poker practice can …
Challenges in trustworthy human evaluation of chatbots
Open community-driven platforms like Chatbot Arena that collect user preference data from
site visitors have gained a reputation as one of the most trustworthy publicly available …
site visitors have gained a reputation as one of the most trustworthy publicly available …
Task design in complex crowdsourcing experiments: Item assignment optimization
Crowdsourcing is the practice of outsourcing a task that would otherwise be performed by
one or a few experts to a crowd of individuals. It is often used to collect large amounts of …
one or a few experts to a crowd of individuals. It is often used to collect large amounts of …
Crowd vs. expert: What can relevance judgment rationales teach us about assessor disagreement?
While crowdsourcing offers a low-cost, scalable way to collect relevance judgments, lack of
transparency with remote crowd work has limited understanding about the quality of …
transparency with remote crowd work has limited understanding about the quality of …
Quality control in crowdsourcing based on fine-grained behavioral features
Crowdsourcing is popular for large-scale data collection and labeling, but a major challenge
is on detecting low-quality submissions. Recent studies have demonstrated that behavioral …
is on detecting low-quality submissions. Recent studies have demonstrated that behavioral …