Quality improvement methods are essential to gathering high-quality crowdsourced data,
both for research and industry applications. A popular and broadly applicable method is task …

With crowdsourcing systems, labels can be obtained with low cost, which facilitates the
creation of training sets for prediction model learning. However, the labels obtained from …

In recent years, the reliability of information on the Internet has emerged as a crucial issue of
modern society. Social network sites (SNSs) have revolutionized the way in which …

Annotation errors and bias are inevitable among different facial expression datasets due to
the subjectiveness of annotating facial expressions. Ascribe to the inconsistent annotations …

Crowdsourcing has emerged as a novel problem-solving paradigm, which facilitates
addressing problems that are hard for computers, eg, entity resolution and sentiment …

Any important data management and analytics tasks cannot be completely addressed by
automated processes. These tasks, such as entity resolution, sentiment analysis, and image …

Data are often labeled by many different experts with each expert only labeling a small
fraction of the data and each data point being labeled by several experts. This reduces the …

Crowdsourcing is a popular paradigm for effectively collecting labels at low cost. The Dawid-
Skene estimator has been widely used for inferring the true labels from the noisy labels …

This survey provides a comprehensive overview of the landscape of crowdsourcing
research, targeted at the machine learning community. We begin with an overview of the …

Label noise in real-world datasets encodes wrong correlation patterns and impairs the
generalization of deep neural networks (DNNs). It is critical to find efficient ways to detect …