Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916264161574912 |
|---|---|
| author | Alizadeh, Meysam Kubli, Maël Samei, Zeynab Dehghani, Shirin Zahedivafa, Mohammadmasiha Bermeo, Juan Diego Korobeynikova, Maria Gilardi, Fabrizio |
| author_facet | Alizadeh, Meysam Kubli, Maël Samei, Zeynab Dehghani, Shirin Zahedivafa, Mohammadmasiha Bermeo, Juan Diego Korobeynikova, Maria Gilardi, Fabrizio |
| contents | This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT-3.5 and GPT-4, though still lagging behind fine-tuned GPT-3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_02179 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning Alizadeh, Meysam Kubli, Maël Samei, Zeynab Dehghani, Shirin Zahedivafa, Mohammadmasiha Bermeo, Juan Diego Korobeynikova, Maria Gilardi, Fabrizio Computation and Language This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT-3.5 and GPT-4, though still lagging behind fine-tuned GPT-3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers. |
| title | Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2307.02179 |