Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2512.05461 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911304084619264 |
|---|---|
| author | Zhang, Bolun Li, Linzhuo Chen, Yunqi Zhao, Qinlin Zhu, Zihan Yi, Xiaoyuan Xie, Xing |
| author_facet | Zhang, Bolun Li, Linzhuo Chen, Yunqi Zhao, Qinlin Zhu, Zihan Yi, Xiaoyuan Xie, Xing |
| contents | Large language models (LLMs) are rapidly being integrated into computational social science research, yet their blackboxed training and designed stochastic elements in inference pose unique challenges for scientific inquiry. This article argues that applying LLMs to social scientific tasks requires explicit assessment of uncertainty-an expectation long established in both quantitative methodology in the social sciences and machine learning. We introduce a unified framework for evaluating LLM uncertainty along two dimensions: the task type (T), which distinguishes between classification, short-form, and long-form generation, and the validation type (V), which captures the availability of reference data or evaluative criteria. Drawing from both computer science and social science literature, we map existing uncertainty quantification (UQ) methods to this T-V typology and offer practical recommendations for researchers. Our framework provides both a methodological safeguard and a practical guide for integrating LLMs into rigorous social science research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_05461 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Knowing Your Uncertainty -- On the application of LLM in social sciences Zhang, Bolun Li, Linzhuo Chen, Yunqi Zhao, Qinlin Zhu, Zihan Yi, Xiaoyuan Xie, Xing Computers and Society Artificial Intelligence Human-Computer Interaction Large language models (LLMs) are rapidly being integrated into computational social science research, yet their blackboxed training and designed stochastic elements in inference pose unique challenges for scientific inquiry. This article argues that applying LLMs to social scientific tasks requires explicit assessment of uncertainty-an expectation long established in both quantitative methodology in the social sciences and machine learning. We introduce a unified framework for evaluating LLM uncertainty along two dimensions: the task type (T), which distinguishes between classification, short-form, and long-form generation, and the validation type (V), which captures the availability of reference data or evaluative criteria. Drawing from both computer science and social science literature, we map existing uncertainty quantification (UQ) methods to this T-V typology and offer practical recommendations for researchers. Our framework provides both a methodological safeguard and a practical guide for integrating LLMs into rigorous social science research. |
| title | Knowing Your Uncertainty -- On the application of LLM in social sciences |
| topic | Computers and Society Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2512.05461 |