Cover Image

Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Abercrombie, Gavin, Dinkar, Tanvi, Curry, Amanda Cercas, Rieser, Verena, Hovy, Dirk
Format:	Preprint
Published:	2023
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution
by: Plaza-del-Arco, Flor Miriam, et al.
Published: (2024)

Subjective $\textit{Isms}$? On the Danger of Conflating Hate and Offence in Abusive Language Detection
by: Curry, Amanda Cercas, et al.
Published: (2024)

Do Large Language Models Adapt to Language Variation across Socioeconomic Status?
by: Bassignana, Elisa, et al.
Published: (2026)

The AI Gap: How Socioeconomic Status Affects Language Technology Interactions
by: Bassignana, Elisa, et al.
Published: (2025)

Impoverished Language Technology: The Lack of (Social) Class in NLP
by: Curry, Amanda Cercas, et al.
Published: (2024)

Re-examining Sexism and Misogyny Classification with Annotator Attitudes
by: Jiang, Aiqi, et al.
Published: (2024)

Erasing 'Ugly' from the Internet: Propagation of the Beauty Myth in Text-Image Models
by: Dinkar, Tanvi, et al.
Published: (2025)

Classist Tools: Social Class Correlates with Performance in NLP
by: Curry, Amanda Cercas, et al.
Published: (2024)

Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions
by: Plaza-del-Arco, Flor Miriam, et al.
Published: (2024)

Divine LLaMAs: Bias, Stereotypes, Stigmatization, and Emotion Representation of Religion in Large Language Models
by: Plaza-del-Arco, Flor Miriam, et al.
Published: (2024)

Food Noise & False Safety: A Systematic Evaluation of How LLMs Fail to Adapt to Eating Disorder Queries with Clinician Feedback
by: Pucci, Giulia, et al.
Published: (2026)

The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics
by: Orlikowski, Matthias, et al.
Published: (2023)

Can NLP Tackle Hate Speech in the Real World? Stakeholder-Informed Feedback and Survey on Counterspeech
by: Dinkar, Tanvi, et al.
Published: (2025)

Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation
by: Plaza-del-Arco, Flor Miriam, et al.
Published: (2023)

NLP Verification: Towards a General Methodology for Certifying Robustness
by: Casadio, Marco, et al.
Published: (2024)

P1SCO: Social Dimensions from a Perspectivist Lens
by: Curry, Amanda Cercas, et al.
Published: (2026)

Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks
by: Zhao, Justin, et al.
Published: (2024)

Compromesso! Italian Many-Shot Jailbreaks Undermine the Safety of Large Language Models
by: Pernisi, Fabio, et al.
Published: (2024)

Conversations as a Source for Teaching Scientific Concepts at Different Education Levels
by: Rooein, Donya, et al.
Published: (2024)

Narratives at Conflict: Computational Analysis of News Framing in Multilingual Disinformation Campaigns
by: Sinelnik, Antonina, et al.
Published: (2024)

From Chatbots to Confidants: A Cross-Cultural Study of LLM Adoption for Emotional Support
by: Amat-Lefort, Natalia, et al.
Published: (2026)

The Call for Socially Aware Language Technologies
by: Yang, Diyi, et al.
Published: (2024)

Interpretable Discriminative Text Representations via Agreement and Label Disentanglement
by: Wang, Tong, et al.
Published: (2026)

NLP for Counterspeech against Hate: A Survey and How-To Guide
by: Bonaldi, Helena, et al.
Published: (2024)

Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings
by: Gonzalez-Gutierrez, Cesar, et al.
Published: (2025)

Are Large Language Models Consistent over Value-laden Questions?
by: Moore, Jared, et al.
Published: (2024)

Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?
by: Ni, Jingwei, et al.
Published: (2025)

EEVEE: An Easy Annotation Tool for Natural Language Processing
by: Sorensen, Axel, et al.
Published: (2024)

The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models
by: Russo, Giuseppe, et al.
Published: (2025)

SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety
by: Röttger, Paul, et al.
Published: (2024)

Estimating Agreement by Chance for Sequence Annotation
by: Li, Diya, et al.
Published: (2024)

PET: An Annotated Dataset for Process Extraction from Natural Language Text
by: Bellan, Patrizio, et al.
Published: (2022)

Exploring Subjective Tasks in Farsi: A Survey Analysis and Evaluation of Language Models
by: Rooein, Donya, et al.
Published: (2025)

Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation
by: Baumann, Joachim, et al.
Published: (2025)

Biased Tales: Cultural and Topic Bias in Generating Children's Stories
by: Rooein, Donya, et al.
Published: (2025)

Beyond Flesch-Kincaid: Prompt-based Metrics Improve Difficulty Classification of Educational Texts
by: Rooein, Donya, et al.
Published: (2024)

Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps
by: Attanasio, Giuseppe, et al.
Published: (2024)

VariErr NLI: Separating Annotation Error from Human Label Variation
by: Weber-Genzel, Leon, et al.
Published: (2024)

Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations
by: Peng, Siyao, et al.
Published: (2024)

What Can Natural Language Processing Do for Peer Review?
by: Kuznetsov, Ilia, et al.
Published: (2024)