Are Expert-Level Language Models Expert-Level Annotators?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912058322190336 |
|---|---|
| author | Tseng, Yu-Min Chen, Wei-Lin Chen, Chung-Chi Chen, Hsin-Hsi |
| author_facet | Tseng, Yu-Min Chen, Wei-Lin Chen, Chung-Chi Chen, Hsin-Hsi |
| contents | Data annotation refers to the labeling or tagging of textual data with relevant information. A large body of works have reported positive results on leveraging LLMs as an alternative to human annotators. However, existing studies focus on classic NLP tasks, and the extent to which LLMs as data annotators perform in domains requiring expert knowledge remains underexplored. In this work, we investigate comprehensive approaches across three highly specialized domains and discuss practical suggestions from a cost-effectiveness perspective. To the best of our knowledge, we present the first systematic evaluation of LLMs as expert-level data annotators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03254 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Are Expert-Level Language Models Expert-Level Annotators? Tseng, Yu-Min Chen, Wei-Lin Chen, Chung-Chi Chen, Hsin-Hsi Computation and Language Data annotation refers to the labeling or tagging of textual data with relevant information. A large body of works have reported positive results on leveraging LLMs as an alternative to human annotators. However, existing studies focus on classic NLP tasks, and the extent to which LLMs as data annotators perform in domains requiring expert knowledge remains underexplored. In this work, we investigate comprehensive approaches across three highly specialized domains and discuss practical suggestions from a cost-effectiveness perspective. To the best of our knowledge, we present the first systematic evaluation of LLMs as expert-level data annotators. |
| title | Are Expert-Level Language Models Expert-Level Annotators? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.03254 |