Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913221341872128 |
|---|---|
| author | Peng, Siyao Sun, Zihang Loftus, Sebastian Plank, Barbara |
| author_facet | Peng, Siyao Sun, Zihang Loftus, Sebastian Plank, Barbara |
| contents | Named Entity Recognition (NER) is a key information extraction task with a long-standing tradition. While recent studies address and aim to correct annotation errors via re-labeling efforts, little is known about the sources of human label variation, such as text ambiguity, annotation error, or guideline divergence. This is especially the case for high-quality datasets and beyond English CoNLL03. This paper studies disagreements in expert-annotated named entity datasets for three languages: English, Danish, and Bavarian. We show that text ambiguity and artificial guideline changes are dominant factors for diverse annotations among high-quality revisions. We survey student annotations on a subset of difficult entities and substantiate the feasibility and necessity of manifold annotations for understanding named entity ambiguities from a distributional perspective. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01423 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations Peng, Siyao Sun, Zihang Loftus, Sebastian Plank, Barbara Computation and Language Named Entity Recognition (NER) is a key information extraction task with a long-standing tradition. While recent studies address and aim to correct annotation errors via re-labeling efforts, little is known about the sources of human label variation, such as text ambiguity, annotation error, or guideline divergence. This is especially the case for high-quality datasets and beyond English CoNLL03. This paper studies disagreements in expert-annotated named entity datasets for three languages: English, Danish, and Bavarian. We show that text ambiguity and artificial guideline changes are dominant factors for diverse annotations among high-quality revisions. We survey student annotations on a subset of difficult entities and substantiate the feasibility and necessity of manifold annotations for understanding named entity ambiguities from a distributional perspective. |
| title | Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.01423 |