Cover Image

A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Levi, Effi, Shenhav, Shaul R.
Format:	Preprint
Published:	2022
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Dialogues of Dissent: Thematic and Rhetorical Dimensions of Hate and Counter-Hate Speech in Social Media Conversations
by: Levi, Effi, et al.
Published: (2025)

Reap the Wild Wind: Detecting Media Storms in Large-Scale News Corpora
by: Markus, Dror K., et al.
Published: (2024)

IsraParlTweet: The Israeli Parliamentary and Twitter Resource
by: Mor-Lan, Guy, et al.
Published: (2024)

You're Not Gonna Believe This: A Computational Analysis of Factual Appeals and Sourcing in Partisan News
by: Mor-Lan, Guy, et al.
Published: (2025)

Exploring Factual Entailment with NLI: A News Media Study
by: Mor-Lan, Guy, et al.
Published: (2024)

FactAppeal: Identifying Epistemic Factual Appeals in News Media
by: Mor-Lan, Guy, et al.
Published: (2025)

HebID: Detecting Social Identities in Hebrew-language Political Text
by: Mor-Lan, Guy, et al.
Published: (2025)

Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP
by: Xu, Yinuo, et al.
Published: (2026)

Leveraging Annotator Disagreement for Text Classification
by: Xu, Jin, et al.
Published: (2024)

NUTMEG: Separating Signal From Noise in Annotator Disagreement
by: Ivey, Jonathan, et al.
Published: (2025)

Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives
by: Xu, Yinuo, et al.
Published: (2025)

Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?
by: Khurana, Urja, et al.
Published: (2024)

Listen to the Layers: Mitigating Hallucinations with Inter-Layer Disagreement
by: Subbalakshmi, Koduvayur, et al.
Published: (2026)

When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks
by: Fleisig, Eve, et al.
Published: (2023)

Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference
by: Kellert, Olga, et al.
Published: (2026)

JuniperLiu at CoMeDi Shared Task: Models as Annotators in Lexical Semantics Disagreements
by: Liu, Zhu, et al.
Published: (2024)

Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?
by: Ni, Jingwei, et al.
Published: (2025)

Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation
by: James, Joseph
Published: (2026)

Dealing with Annotator Disagreement in Hate Speech Classification
by: Dehghan, Somaiyeh, et al.
Published: (2025)

Decoding Climate Disagreement: A Graph Neural Network-Based Approach to Understanding Social Media Dynamics
by: Su, Ruiran, et al.
Published: (2024)

Funzac at CoMeDi Shared Task: Modeling Annotator Disagreement from Word-In-Context Perspectives
by: Sarumi, Olufunke O., et al.
Published: (2025)

Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement
by: Lu, Junyu, et al.
Published: (2025)

Enhancing LLM-Based Data Annotation with Error Decomposition
by: Xu, Zhen, et al.
Published: (2026)

Analyzing Dataset Annotation Quality Management in the Wild
by: Klie, Jan-Christoph, et al.
Published: (2023)

Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation
by: Maurer, Maximilian, et al.
Published: (2026)

When Disagreements Elicit Robustness: Investigating Self-Repair Capabilities under LLM Multi-Agent Disagreements
by: Ju, Tianjie, et al.
Published: (2025)

The Media Bias Detector: A Framework for Annotating and Analyzing the News at Scale
by: Haider, Samar, et al.
Published: (2025)

Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement
by: Hirota, Wataru, et al.
Published: (2026)

D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation
by: Davani, Aida Mostafazadeh, et al.
Published: (2024)

Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation
by: Kocmi, Tom, et al.
Published: (2024)

Quantifying and Predicting Disagreement in Graded Human Ratings
by: Zhang, Leixin, et al.
Published: (2026)

A Dataset for Physical and Abstract Plausibility and Sources of Human Disagreement
by: Eichel, Annerose, et al.
Published: (2024)

Understanding and Analyzing Inappropriately Targeting Language in Online Discourse: A Comparative Annotation Study
by: Barbarestani, Baran, et al.
Published: (2025)

Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis
by: Lu, Junyu, et al.
Published: (2026)

Do Differences in Values Influence Disagreements in Online Discussions?
by: van der Meer, Michiel, et al.
Published: (2023)

Bridging the Gap: In-Context Learning for Modeling Human Disagreement
by: Muscato, Benedetta, et al.
Published: (2025)

From Disagreement to Understanding: The Case for Ambiguity Detection in NLI
by: Jayaweera, Chathuri, et al.
Published: (2025)

Med-CoDE: Medical Critique based Disagreement Evaluation Framework
by: Gupta, Mohit, et al.
Published: (2025)

Disagreement as Data: Reasoning Trace Analytics in Multi-Agent Systems
by: Tajik, Elham, et al.
Published: (2026)

Benchmark Illusion: Disagreement among LLMs and Its Scientific Consequences
by: Yang, Eddie, et al.
Published: (2026)