Abstract We introduce Semi-supervised Performance Evaluation for Face Recognition (SPE-
FR). SPE-FR is a statistical method for evaluating the performance and algorithmic bias of …

In crowdsourcing systems, requesters publish tasks, and interested workers provide
answers to get rewards. Worker anonymity motivates participation since it protects their …

In this paper, we present a comprehensive study that evaluates six state-of-the-art sentiment
analysis tools on five public datasets, based on the quality of predictive results in the …

The introduction of smart building technology promises many operational and productivity
benefits and enables smart grid integration. A significant barrier to deploying smart building …

Crowdsourcing has emerged as an effective platform for labeling large amounts of data in a
cost-and time-efficient manner. Most previous work has focused on designing an efficient …

We introduce a learning framework for the problem of unifying conflicting data in multiple
records referring to the same entity—we call this problem “record fusion.” Record fusion …

Citizen science has become a valuable and reliable method for interpreting and processing
big datasets, and is vital in the era of ever-growing data volumes. However, there are …

In crowdsourcing systems, requesters publish tasks, and interested workers provide
answers to get rewards. Worker anonymity motivates participation since it protects their …

We propose a novel three-stage FIND-RESOLVE-LABEL workflow for crowdsourced
annotation to reduce ambiguity in task instructions and, thus, improve annotation quality …

''Easy/hard sample” is a popular parlance in machine learning. Learning difficulty of samples
refers to how easy/hard a sample is during a learning procedure. An increasing need of …