Abstract Many tasks in Natural Language Processing (NLP) and Computer Vision (CV) offer
evidence that humans disagree, from objective tasks such as part-of-speech tagging to more …

Recent research has demonstrated that cognitive biases such as the confirmation bias or the
anchoring effect can negatively affect the quality of crowdsourced data. In practice, however …

Relation extraction (RE) is an essential topic in natural language processing and has
attracted extensive attention. Current RE approaches achieve fantastic results on common …

When crowdsourcing the creation of machine learning datasets, statistical distributions that
capture diverse answers can represent ambiguous data better than a single best answer …

Social media have the potential to provide timely information about emergency situations
and sudden events. However, finding relevant information among the millions of posts being …

In many linguistic fields requiring annotated data, multiple interpretations of a single item are
possible. Multi-label annotations more accurately reflect this possibility. However, allowing …

As NLP systems become better at detecting opinions and beliefs from text, it is important to
ensure not only that models are accurate but also that they arrive at their predictions in ways …

The rapid entry of machine learning approaches in our daily activities and high-stakes
domains demands transparency and scrutiny of their fairness and reliability. To help gauge …

Asking annotators to explain “why” they labeled an instance yields annotator rationales:
natural language explanations that provide reasons for classifications. In this work, we …

In most classification models, it has been assumed to have a single ground truth label for
each data point. However, subjective tasks like toxicity classification can lead to genuine …