Evaluating the Impact of LLM-Assisted Annotation in a Perspectivized Setting: the Case of FrameNet Annotation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914123714920448 |
|---|---|
| author | Belcavello, Frederico Matos, Ely Lorenzi, Arthur Bonoto, Lisandra Ruiz, Lívia Pereira, Luiz Fernando Herbst, Victor Navarro, Yulla Abreu, Helen de Andrade Dutra, Lívia Torrent, Tiago Timponi |
| author_facet | Belcavello, Frederico Matos, Ely Lorenzi, Arthur Bonoto, Lisandra Ruiz, Lívia Pereira, Luiz Fernando Herbst, Victor Navarro, Yulla Abreu, Helen de Andrade Dutra, Lívia Torrent, Tiago Timponi |
| contents | The use of LLM-based applications as a means to accelerate and/or substitute human labor in the creation of language resources and dataset is a reality. Nonetheless, despite the potential of such tools for linguistic research, comprehensive evaluation of their performance and impact on the creation of annotated datasets, especially under a perspectivized approach to NLP, is still missing. This paper contributes to reduction of this gap by reporting on an extensive evaluation of the (semi-)automatization of FrameNet-like semantic annotation by the use of an LLM-based semantic role labeler. The methodology employed compares annotation time, coverage and diversity in three experimental settings: manual, automatic and semi-automatic annotation. Results show that the hybrid, semi-automatic annotation setting leads to increased frame diversity and similar annotation coverage, when compared to the human-only setting, while the automatic setting performs considerably worse in all metrics, except for annotation time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_25904 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating the Impact of LLM-Assisted Annotation in a Perspectivized Setting: the Case of FrameNet Annotation Belcavello, Frederico Matos, Ely Lorenzi, Arthur Bonoto, Lisandra Ruiz, Lívia Pereira, Luiz Fernando Herbst, Victor Navarro, Yulla Abreu, Helen de Andrade Dutra, Lívia Torrent, Tiago Timponi Computation and Language Artificial Intelligence The use of LLM-based applications as a means to accelerate and/or substitute human labor in the creation of language resources and dataset is a reality. Nonetheless, despite the potential of such tools for linguistic research, comprehensive evaluation of their performance and impact on the creation of annotated datasets, especially under a perspectivized approach to NLP, is still missing. This paper contributes to reduction of this gap by reporting on an extensive evaluation of the (semi-)automatization of FrameNet-like semantic annotation by the use of an LLM-based semantic role labeler. The methodology employed compares annotation time, coverage and diversity in three experimental settings: manual, automatic and semi-automatic annotation. Results show that the hybrid, semi-automatic annotation setting leads to increased frame diversity and similar annotation coverage, when compared to the human-only setting, while the automatic setting performs considerably worse in all metrics, except for annotation time. |
| title | Evaluating the Impact of LLM-Assisted Annotation in a Perspectivized Setting: the Case of FrameNet Annotation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.25904 |