Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909152263012352 |
|---|---|
| author | Aguda, Toyin Siddagangappa, Suchetha Kochkina, Elena Kaur, Simerjot Wang, Dongsheng Smiley, Charese Shah, Sameena |
| author_facet | Aguda, Toyin Siddagangappa, Suchetha Kochkina, Elena Kaur, Simerjot Wang, Dongsheng Smiley, Charese Shah, Sameena |
| contents | Collecting labeled datasets in finance is challenging due to scarcity of domain experts and higher cost of employing them. While Large Language Models (LLMs) have demonstrated remarkable performance in data annotation tasks on general domain datasets, their effectiveness on domain specific datasets remains underexplored. To address this gap, we investigate the potential of LLMs as efficient data annotators for extracting relations in financial documents. We compare the annotations produced by three LLMs (GPT-4, PaLM 2, and MPT Instruct) against expert annotators and crowdworkers. We demonstrate that the current state-of-the-art LLMs can be sufficient alternatives to non-expert crowdworkers. We analyze models using various prompts and parameter settings and find that customizing the prompts for each relation group by providing specific examples belonging to those groups is paramount. Furthermore, we introduce a reliability index (LLM-RelIndex) used to identify outputs that may require expert attention. Finally, we perform an extensive time, cost and error analysis and provide recommendations for the collection and usage of automated annotations in domain-specific settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_18152 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency Aguda, Toyin Siddagangappa, Suchetha Kochkina, Elena Kaur, Simerjot Wang, Dongsheng Smiley, Charese Shah, Sameena Computation and Language Collecting labeled datasets in finance is challenging due to scarcity of domain experts and higher cost of employing them. While Large Language Models (LLMs) have demonstrated remarkable performance in data annotation tasks on general domain datasets, their effectiveness on domain specific datasets remains underexplored. To address this gap, we investigate the potential of LLMs as efficient data annotators for extracting relations in financial documents. We compare the annotations produced by three LLMs (GPT-4, PaLM 2, and MPT Instruct) against expert annotators and crowdworkers. We demonstrate that the current state-of-the-art LLMs can be sufficient alternatives to non-expert crowdworkers. We analyze models using various prompts and parameter settings and find that customizing the prompts for each relation group by providing specific examples belonging to those groups is paramount. Furthermore, we introduce a reliability index (LLM-RelIndex) used to identify outputs that may require expert attention. Finally, we perform an extensive time, cost and error analysis and provide recommendations for the collection and usage of automated annotations in domain-specific settings. |
| title | Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.18152 |