Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916344588402688 |
|---|---|
| author | Møller, Anders Giovanni Aiello, Luca Maria |
| author_facet | Møller, Anders Giovanni Aiello, Luca Maria |
| contents | Large Language Models are expressive tools that enable complex tasks of text understanding within Computational Social Science. Their versatility, while beneficial, poses a barrier for establishing standardized best practices within the field. To bring clarity on the values of different strategies, we present an overview of the performance of modern LLM-based classification methods on a benchmark of 23 social knowledge tasks. Our results point to three best practices: select models with larger vocabulary and pre-training corpora; avoid simple zero-shot in favor of AI-enhanced prompting; fine-tune on task-specific data, and consider more complex forms instruction-tuning on multiple datasets only when only training data is more abundant. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01346 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks Møller, Anders Giovanni Aiello, Luca Maria Computers and Society Computation and Language Physics and Society Large Language Models are expressive tools that enable complex tasks of text understanding within Computational Social Science. Their versatility, while beneficial, poses a barrier for establishing standardized best practices within the field. To bring clarity on the values of different strategies, we present an overview of the performance of modern LLM-based classification methods on a benchmark of 23 social knowledge tasks. Our results point to three best practices: select models with larger vocabulary and pre-training corpora; avoid simple zero-shot in favor of AI-enhanced prompting; fine-tune on task-specific data, and consider more complex forms instruction-tuning on multiple datasets only when only training data is more abundant. |
| title | Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks |
| topic | Computers and Society Computation and Language Physics and Society |
| url | https://arxiv.org/abs/2408.01346 |