Simple, robust and optimal ranking from pairwise comparisons

NB Shah, MJ Wainwright - Journal of machine learning research, 2018‏ - jmlr.org
We consider data in the form of pairwise comparisons of n items, with the goal of identifying
the top k items for some value of k< n, or alternatively, recovering a ranking of all the items …

Exploiting worker correlation for label aggregation in crowdsourcing

Y Li, B Rubinstein, T Cohn - International conference on …, 2019‏ - proceedings.mlr.press
Crowdsourcing has emerged as a core component of data science pipelines. From collected
noisy worker labels, aggregation models that incorporate worker reliability parameters aim …

Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings

J Wang, NB Shah - arxiv preprint arxiv:1806.05085, 2018‏ - arxiv.org
Cardinal scores (numeric ratings) collected from people are well known to suffer from
miscalibrations. A popular approach to address this issue is to assume simplistic models of …

Double or nothing: Multiplicative incentive mechanisms for crowdsourcing

NB Shah, D Zhou - Journal of Machine Learning Research, 2016‏ - jmlr.org
Crowdsourcing has gained immense popularity in machine learning applications for
obtaining large amounts of labeled data. Crowdsourcing is cheap and fast, but suffers from …

Learning from crowds by modeling common confusions

Z Chu, J Ma, H Wang - Proceedings of the AAAI Conference on Artificial …, 2021‏ - ojs.aaai.org
Crowdsourcing provides a practical way to obtain large amounts of labeled data at a low
cost. However, the annotation quality of annotators varies considerably, which imposes new …

Achieving budget-optimality with adaptive schemes in crowdsourcing

A Khetan, S Oh - Advances in Neural Information …, 2016‏ - proceedings.neurips.cc
Adaptive schemes, where tasks are assigned based on the data collected thus far, are
widely used in practical crowdsourcing systems to efficiently allocate the budget. However …

Max-mig: an information theoretic approach for joint learning from crowds

P Cao, Y Xu, Y Kong, Y Wang - arxiv preprint arxiv:1905.13436, 2019‏ - arxiv.org
Eliciting labels from crowds is a potential way to obtain large labeled data. Despite a variety
of methods developed for learning from crowds, a key challenge remains unsolved:\emph …

Isotonic regression with unknown permutations: Statistics, computation and adaptation

A Pananjady, RJ Samworth - The Annals of Statistics, 2022‏ - projecteuclid.org
Isotonic regression with unknown permutations: Statistics, computation and adaptation Page 1
The Annals of Statistics 2022, Vol. 50, No. 1, 324–350 https://doi.org/10.1214/21-AOS2107 © …

Approval voting and incentives in crowdsourcing

NB Shah, D Zhou - ACM Transactions on Economics and Computation …, 2020‏ - dl.acm.org
The growing need for labeled training data has made crowdsourcing a vital tool for
develo** machine learning applications. Here, workers on a crowdsourcing platform are …

Adversarial crowdsourcing through robust rank-one matrix completion

Q Ma, A Olshevsky - Advances in Neural Information …, 2020‏ - proceedings.neurips.cc
We consider the problem of reconstructing a rank-one matrix from a revealed subset of its
entries when some of the revealed entries are corrupted with perturbations that are unknown …