Crowdworksheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation

M Díaz, I Kivlichan, R Rosen, D Baker… - Proceedings of the …, 2022 - dl.acm.org
Human annotated data plays a crucial role in machine learning (ML) research and
development. However, the ethical considerations around the processes and decisions that …

Hate speech classifiers learn normative social stereotypes

AM Davani, M Atari, B Kennedy… - Transactions of the …, 2023 - direct.mit.edu
Social stereotypes negatively impact individuals' judgments about different groups and may
have a critical role in understanding language directed toward marginalized groups. Here …

Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation

N Goyal, ID Kivlichan, R Rosen… - Proceedings of the ACM …, 2022 - dl.acm.org
Machine learning models are commonly used to detect toxicity in online conversations.
These models are trained on datasets annotated by human raters. We explore how raters' …

The PRISM alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large …

HR Kirk, A Whitefield, P Rottger… - Advances in …, 2025 - proceedings.neurips.cc
Human feedback is central to the alignment of Large Language Models (LLMs). However,
open questions remain about the methods (how), domains (where), people (who) and …

Toxicity detection: Does context really matter?

J Pavlopoulos, J Sorensen, L Dixon, N Thain… - arxiv preprint arxiv …, 2020 - arxiv.org
Moderation is crucial to promoting healthy on-line discussions. Although severaltoxicity'
detection datasets and models have been published, most of them ignore the context of the …

Commonality and variation in mental representations of music revealed by a cross-cultural comparison of rhythm priors in 15 countries

N Jacoby, R Polak, JA Grahn, DJ Cameron… - Nature Human …, 2024 - nature.com
Music is present in every known society but varies from place to place. What, if anything, is
universal to music cognition? We measured a signature of mental representations of rhythm …

Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts

L Breitfeller, E Ahn, D Jurgens… - Proceedings of the 2019 …, 2019 - aclanthology.org
Microaggressions are subtle, often veiled, manifestations of human biases. These uncivil
interactions can have a powerful negative impact on people by marginalizing minorities and …

Whose ground truth? accounting for individual and collective identities underlying dataset annotation

R Denton, M Díaz, I Kivlichan, V Prabhakaran… - arxiv preprint arxiv …, 2021 - arxiv.org
Human annotations play a crucial role in machine learning (ML) research and development.
However, the ethical considerations around the processes and decisions that go into …

Convenience samples in political science experiments

Y Krupnikov, HH Nam, H Style - Advances in experimental …, 2021 - books.google.com
This chapter provides an overview of convenience sampling in political science
experimental research. We first define convenience sampling and track its use in political …

Frustration: still a common user experience

M Hertzum, K Hornbæk - ACM Transactions on Computer-Human …, 2023 - dl.acm.org
When computers unexpectedly delay or thwart goal attainment, frustration ensues. The
central studies of the extent, content, and impact of such frustration were done more than 15 …