Arti-"fickle" Intelligence: Using LLMs as a Tool for Inference in the Political and Social Sciences
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910904195481600 |
|---|---|
| author | Argyle, Lisa P. Busby, Ethan C. Gubler, Joshua R. Hepner, Bryce Lyman, Alex Wingate, David |
| author_facet | Argyle, Lisa P. Busby, Ethan C. Gubler, Joshua R. Hepner, Bryce Lyman, Alex Wingate, David |
| contents | Generative large language models (LLMs) are incredibly useful, versatile, and promising tools. However, they will be of most use to political and social science researchers when they are used in a way that advances understanding about real human behaviors and concerns. To promote the scientific use of LLMs, we suggest that researchers in the political and social sciences need to remain focused on the scientific goal of inference. To this end, we discuss the challenges and opportunities related to scientific inference with LLMs, using validation of model output as an illustrative case for discussion. We propose a set of guidelines related to establishing the failure and success of LLMs when completing particular tasks, and discuss how we can make inferences from these observations. We conclude with a discussion of how this refocus will improve the accumulation of shared scientific knowledge about these tools and their uses in the social sciences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_03822 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Arti-"fickle" Intelligence: Using LLMs as a Tool for Inference in the Political and Social Sciences Argyle, Lisa P. Busby, Ethan C. Gubler, Joshua R. Hepner, Bryce Lyman, Alex Wingate, David Computers and Society Artificial Intelligence Generative large language models (LLMs) are incredibly useful, versatile, and promising tools. However, they will be of most use to political and social science researchers when they are used in a way that advances understanding about real human behaviors and concerns. To promote the scientific use of LLMs, we suggest that researchers in the political and social sciences need to remain focused on the scientific goal of inference. To this end, we discuss the challenges and opportunities related to scientific inference with LLMs, using validation of model output as an illustrative case for discussion. We propose a set of guidelines related to establishing the failure and success of LLMs when completing particular tasks, and discuss how we can make inferences from these observations. We conclude with a discussion of how this refocus will improve the accumulation of shared scientific knowledge about these tools and their uses in the social sciences. |
| title | Arti-"fickle" Intelligence: Using LLMs as a Tool for Inference in the Political and Social Sciences |
| topic | Computers and Society Artificial Intelligence |
| url | https://arxiv.org/abs/2504.03822 |