Human variation in labeling is often considered noise. Annotation projects for machine
learning (ML) aim at minimizing human label variation, with the assumption to maximize …

Semantic processing is a fundamental research domain in computational linguistics. In the
era of powerful pre-trained language models and large language models, the advancement …

Social stereotypes negatively impact individuals' judgments about different groups and may
have a critical role in understanding language directed toward marginalized groups. Here …

We investigate how disagreement in natural language inference (NLI) annotation arises. We
developed a taxonomy of disagreement sources with 10 categories spanning 3 high-level …

High-quality data is necessary for modern machine learning. However, the acquisition of
such data is difficult due to noisy and ambiguous annotations of humans. The aggregation of …

Annotators' disagreement in linguistic data has been recently the focus of multiple initiatives
aimed at raising awareness on issues related to 'majority voting'when aggregating diverging …

NLP datasets annotated with human judgments are rife with disagreements between the
judges. This is especially true for tasks depending on subjective judgments such as …

The task of fact-checking deals with assessing the veracity of factual claims based on
credible evidence and background knowledge. In particular, scientific fact-checking is the …

The use of misogynistic and sexist language has increased in recent years in social media,
and is increasing in the Arabic world in reaction to reforms attempting to remove restrictions …

Annotators are not fungible. Their demographics, life experiences, and backgrounds all
contribute to how they label data. However, NLP has only recently considered how …