Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917441110540288 |
|---|---|
| author | Gu, Feng Li, Zongxia Colon, Carlos Rafael Evans, Benjamin Mondal, Ishani Boyd-Graber, Jordan Lee |
| author_facet | Gu, Feng Li, Zongxia Colon, Carlos Rafael Evans, Benjamin Mondal, Ishani Boyd-Graber, Jordan Lee |
| contents | Event annotation is important for identifying market changes, monitoring breaking news, and understanding sociological trends. Although expert annotators set the gold standards, human coding is expensive and inefficient. Unlike information extraction experiments that focus on single contexts, we evaluate a holistic workflow that removes irrelevant documents, merges documents about the same event, and annotates the events. Although LLM-based automated annotations are better than traditional TF-IDF-based methods or Event Set Curation, they are still not reliable annotators compared to human experts. However, adding LLMs to assist experts for Event Set Curation can reduce the time and mental effort required for Variable Annotation. When using LLMs to extract event variables to assist expert annotators, they agree more with the extracted variables than fully automated LLMs for annotation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06778 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators Gu, Feng Li, Zongxia Colon, Carlos Rafael Evans, Benjamin Mondal, Ishani Boyd-Graber, Jordan Lee Computation and Language Artificial Intelligence Event annotation is important for identifying market changes, monitoring breaking news, and understanding sociological trends. Although expert annotators set the gold standards, human coding is expensive and inefficient. Unlike information extraction experiments that focus on single contexts, we evaluate a holistic workflow that removes irrelevant documents, merges documents about the same event, and annotates the events. Although LLM-based automated annotations are better than traditional TF-IDF-based methods or Event Set Curation, they are still not reliable annotators compared to human experts. However, adding LLMs to assist experts for Event Set Curation can reduce the time and mental effort required for Variable Annotation. When using LLMs to extract event variables to assist expert annotators, they agree more with the extracted variables than fully automated LLMs for annotation. |
| title | Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.06778 |