Ambiguity is an intrinsic feature of natural language. Managing ambiguity is a key part of
human language understanding, allowing us to anticipate misunderstanding as …

We investigate how disagreement in natural language inference (NLI) annotation arises. We
developed a taxonomy of disagreement sources with 10 categories spanning 3 high-level …

Calibration is a popular framework to evaluate whether a classifier knows when it does not
know-ie, its predictive probabilities are a good indication of how likely a prediction is to be …

This paper presents the results of the SHROOM, a shared task focused on detecting
hallucinations: outputs from natural language generation (NLG) systems that are fluent, yet …

Annotators' sociodemographic backgrounds (ie, the individual compositions of their gender,
age, educational background, etc.) have a strong impact on their decisions when working on …

Training NLP systems typically assumes access to annotated data that has a single human
label per example. Given imperfect labeling from annotators and inherent ambiguity of …

Annotator disagreement is ubiquitous in natural language processing (NLP) tasks. There are
multiple reasons for such disagreements, including the subjectivity of the task, difficult cases …

The model editing problem concerns how language models should learn new facts about
the world over time. While empirical research on model editing has drawn widespread …

Chatbots, the automation of communicative labor, have been widely deployed in industrial
applications and systems. Built upon the Generative Pre-trained Transformer 3 (GPT-3) …

Human label variation (HLV) is a valuable source of information that arises when multiple
human annotators provide different labels for valid reasons. In Natural Language Inference …