LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914846238310400 |
|---|---|
| author | Kholodna, Nataliia Julka, Sahib Khodadadi, Mohammad Gumus, Muhammed Nurullah Granitzer, Michael |
| author_facet | Kholodna, Nataliia Julka, Sahib Khodadadi, Mohammad Gumus, Muhammed Nurullah Granitzer, Michael |
| contents | Low-resource languages face significant barriers in AI development due to limited linguistic resources and expertise for data labeling, rendering them rare and costly. The scarcity of data and the absence of preexisting tools exacerbate these challenges, especially since these languages may not be adequately represented in various NLP datasets. To address this gap, we propose leveraging the potential of LLMs in the active learning loop for data annotation. Initially, we conduct evaluations to assess inter-annotator agreement and consistency, facilitating the selection of a suitable LLM annotator. The chosen annotator is then integrated into a training loop for a classifier using an active learning paradigm, minimizing the amount of queried data required. Empirical evaluations, notably employing GPT-4-Turbo, demonstrate near-state-of-the-art performance with significantly reduced data requirements, as indicated by estimated potential cost savings of at least 42.45 times compared to human annotation. Our proposed solution shows promising potential to substantially reduce both the monetary and computational costs associated with automation in low-resource settings. By bridging the gap between low-resource languages and AI, this approach fosters broader inclusion and shows the potential to enable automation across diverse linguistic landscapes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_02261 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages Kholodna, Nataliia Julka, Sahib Khodadadi, Mohammad Gumus, Muhammed Nurullah Granitzer, Michael Computation and Language Artificial Intelligence Information Retrieval Machine Learning I.2.7; I.2.6 Low-resource languages face significant barriers in AI development due to limited linguistic resources and expertise for data labeling, rendering them rare and costly. The scarcity of data and the absence of preexisting tools exacerbate these challenges, especially since these languages may not be adequately represented in various NLP datasets. To address this gap, we propose leveraging the potential of LLMs in the active learning loop for data annotation. Initially, we conduct evaluations to assess inter-annotator agreement and consistency, facilitating the selection of a suitable LLM annotator. The chosen annotator is then integrated into a training loop for a classifier using an active learning paradigm, minimizing the amount of queried data required. Empirical evaluations, notably employing GPT-4-Turbo, demonstrate near-state-of-the-art performance with significantly reduced data requirements, as indicated by estimated potential cost savings of at least 42.45 times compared to human annotation. Our proposed solution shows promising potential to substantially reduce both the monetary and computational costs associated with automation in low-resource settings. By bridging the gap between low-resource languages and AI, this approach fosters broader inclusion and shows the potential to enable automation across diverse linguistic landscapes. |
| title | LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning I.2.7; I.2.6 |
| url | https://arxiv.org/abs/2404.02261 |