Cover Image

Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gu, Feng, Li, Zongxia, Colon, Carlos Rafael, Evans, Benjamin, Mondal, Ishani, Boyd-Graber, Jordan Lee
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

PEDANTS: Cheap but Effective and Interpretable Answer Equivalence
by: Li, Zongxia, et al.
Published: (2024)

SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity
by: Mondal, Ishani, et al.
Published: (2025)

How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generation
by: Sung, Yoo Yeon, et al.
Published: (2024)

CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert Judgments For Open-Domain Question Answering
by: Li, Zongxia, et al.
Published: (2024)

SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinement
by: Mondal, Ishani, et al.
Published: (2024)

Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness
by: Sung, Yoo Yeon, et al.
Published: (2024)

Large Language Models Struggle to Describe the Haystack without Human Help: Human-in-the-loop Evaluation of Topic Models
by: Li, Zongxia, et al.
Published: (2025)

ANAH: Analytical Annotation of Hallucinations in Large Language Models
by: Ji, Ziwei, et al.
Published: (2024)

Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA
by: Gor, Maharshi, et al.
Published: (2024)

Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?
by: Ni, Jingwei, et al.
Published: (2025)

CANVAS: Continuity-Aware Narratives via Visual Agentic Storyboarding
by: Mondal, Ishani, et al.
Published: (2026)

ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models
by: Gu, Yuzhe, et al.
Published: (2024)

RadAnnotate: Large Language Models for Efficient and Reliable Radiology Report Annotation
by: Shetty, Saisha Pradeep, et al.
Published: (2026)

Astro-NER -- Astronomy Named Entity Recognition: Is GPT a Good Domain Expert Annotator?
by: Evans, Julia, et al.
Published: (2024)

Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025
by: Kunilovskaya, Maria, et al.
Published: (2026)

Entity Alignment with Noisy Annotations from Large Language Models
by: Chen, Shengyuan, et al.
Published: (2024)

Are Large Language Models Reliable Argument Quality Annotators?
by: Mirzakhmedova, Nailia, et al.
Published: (2024)

Are LLMs Good Annotators for Discourse-level Event Relation Extraction?
by: Wei, Kangda, et al.
Published: (2024)

DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering
by: Srikanth, Neha, et al.
Published: (2026)

Investigating Annotator Bias in Large Language Models for Hate Speech Detection
by: Das, Amit, et al.
Published: (2024)

Standardizing Longitudinal Radiology Report Evaluation via Large Language Model Annotation
by: Wang, Xinyi, et al.
Published: (2026)

Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations
by: Ki, Dayeon, et al.
Published: (2024)

Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation
by: Törnberg, Petter
Published: (2026)

Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrong
by: Si, Chenglei, et al.
Published: (2023)

AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models
by: Zbeeb, Mohammad, et al.
Published: (2025)

HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages
by: Wang, Zhilin, et al.
Published: (2025)

"All that Glitters": Approaches to Evaluations with Unreliable Model and Human Annotations
by: Hardy, Michael
Published: (2024)

Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above
by: Balepur, Nishant, et al.
Published: (2025)

Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models
by: Cory, Thomas, et al.
Published: (2025)

GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration
by: Sung, Yoo Yeon, et al.
Published: (2025)

NAVIG: Natural Language-guided Analysis with Vision Language Models for Image Geo-localization
by: Zhang, Zheyuan, et al.
Published: (2025)

AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?
by: Gor, Maharshi, et al.
Published: (2026)

A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users
by: Balepur, Nishant, et al.
Published: (2025)

Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm
by: Bojic, Ljubisa, et al.
Published: (2025)

Annotation Guidelines-Based Knowledge Augmentation: Towards Enhancing Large Language Models for Educational Text Classification
by: Liu, Shiqi, et al.
Published: (2024)

Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation
by: Baumann, Joachim, et al.
Published: (2025)

Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models
by: Abraham, Louis, et al.
Published: (2024)

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
by: Li, Minzhi, et al.
Published: (2023)

Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets
by: Giorgi, Tommaso, et al.
Published: (2024)

Are Large Language Models Good Statisticians?
by: Zhu, Yizhang, et al.
Published: (2024)