Cover Image

The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Munir, Sheza, Mah, Benjamin, Kalsi, Krisha, Kapania, Shivani, Posada, Julian, Law, Edith, Wang, Ding, Ahmed, Syed Ishtiaque
Format:	Preprint
Published:	2026
Subjects:	Artificial Intelligence Computation and Language Computers and Society
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Are You the A-hole? A Fair, Multi-Perspective Ethical Reasoning Framework
by: Munir, Sheza, et al.
Published: (2026)

Prediction Laundering: The Illusion of Neutrality, Transparency, and Governance in Polymarket
by: Rohanifar, Yasaman, et al.
Published: (2026)

Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline
by: Kapania, Shivani, et al.
Published: (2025)

'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants
by: Kapania, Shivani, et al.
Published: (2024)

BTPD: A Multilingual Hand-curated Dataset of Bengali Transnational Political Discourse Across Online Communities
by: Das, Dipto, et al.
Published: (2025)

Consensus and Subjectivity of Skin Tone Annotation for ML Fairness
by: Schumann, Candice, et al.
Published: (2023)

FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation
by: Bayat, Farima Fatahi, et al.
Published: (2024)

When Cow Urine Cures Constipation on YouTube: Limits of LLMs in Detecting Culture-specific Health Misinformation
by: Khan, Anamta, et al.
Published: (2026)

Dharma, Data and Deception: An LLM-Powered Rhetorical Analysis of Cow-Urine Health Claims on YouTube
by: Munir, Sheza, et al.
Published: (2026)

Ethical Analysis on the Application of Neurotechnology for Human Augmentation in Physicians and Surgeons
by: Hossain, Soaad, et al.
Published: (2020)

Ethical Artificial Intelligence Principles and Guidelines for the Governance and Utilization of Highly Advanced Large Language Models
by: Hossain, Soaad, et al.
Published: (2023)

User Detection and Response Patterns of Sycophantic Behavior in Conversational AI
by: Noshin, Kazi, et al.
Published: (2026)

VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts
by: Liu, Xin, et al.
Published: (2025)

"I'm categorizing LLM as a productivity tool": Examining ethics of LLM use in HCI research practices
by: Kapania, Shivani, et al.
Published: (2024)

Bureaucratic Silences: What the Canadian AI Register Reveals, Omits, and Obscures
by: Das, Dipto, et al.
Published: (2026)

Data Repair
by: Rahman, ATM Mizanur, et al.
Published: (2026)

Towards a Non-Ideal Methodological Framework for Responsible ML
by: Mothilal, Ramaravind Kommiya, et al.
Published: (2024)

Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge
by: Song, Mingyang, et al.
Published: (2026)

Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation
by: Liao, Hsuan Wei, et al.
Published: (2025)

Beyond Agreement: Rethinking Ground Truth in Educational AI Annotation
by: Thomas, Danielle R., et al.
Published: (2025)

How do the Global South Diasporas Mobilize for Transnational Political Change?
by: Das, Dipto, et al.
Published: (2026)

Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistency
by: Xu, Haoming, et al.
Published: (2026)

Argument-Based Consistency in Toxicity Explanations of LLMs
by: Mothilal, Ramaravind Kommiya, et al.
Published: (2025)

GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Models
by: Cudlenco, Nicolae, et al.
Published: (2026)

Embodying Facts, Figures, and Faiths in Narrative Artistic Performances in Rural Bangladesh
by: Sultana, Sharifa, et al.
Published: (2026)

Reasoning About Reasoning: Towards Informed and Reflective Use of LLM Reasoning in HCI
by: Mothilal, Ramaravind Kommiya, et al.
Published: (2025)

A Civics-oriented Approach to Understanding Intersectionally Marginalized Users' Experience with Hate Speech Online
by: Sultana, Achhiya, et al.
Published: (2024)

Job Anxiety in Post-Secondary Computer Science Students Caused by Artificial Intelligence
by: Farooqi, Daniyaal, et al.
Published: (2026)

TruthStance: An Annotated Dataset of Conversations on Truth Social
by: Ameen, Fathima, et al.
Published: (2026)

3D Ground Truth Reconstruction from Multi-Camera Annotations Using UKF
by: Van Ma, Linh, et al.
Published: (2025)

Situated Ground Truths: Enhancing Bias-Aware AI by Situating Data Labels with SituAnnotate
by: Pandiani, Delfina Sol Martinez, et al.
Published: (2024)

An Experiential Approach to AI Literacy
by: Khandwaha, Aakanksha, et al.
Published: (2026)

Dissecting Generalized Category Discovery: Multiplex Consensus under Self-Deconstruction
by: Tang, Luyao, et al.
Published: (2025)

Deeply Embedded Wages: Navigating Digital Payments in Data Work
by: Posada, Julian
Published: (2024)

Learning Annotation Consensus for Continuous Emotion Recognition
by: Shoer, Ibrahim, et al.
Published: (2025)

IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth
by: Islam, Md Touhidul, et al.
Published: (2025)

Marine Snow Removal Using Internally Generated Pseudo Ground Truth
by: Malyugina, Alexandra, et al.
Published: (2025)

The Politics of Fear and the Experience of Bangladeshi Religious Minority Communities Using Social Media Platforms
by: Rifat, Mohammad Rashidujjaman, et al.
Published: (2024)

Truth-Revealing Participatory Budgeting
by: Han, Qishen, et al.
Published: (2026)

Building a "-Sensitive Design" Methodology from Political Philosophies or Ideologies
by: Maocheia-Ricci, Anthony, et al.
Published: (2026)