Selecting Language Models for Social Science: Start Small, Start Open, and Validate
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908769541160960 |
|---|---|
| author | Stoltz, Dustin S. Taylor, Marshall A. Kumar, Sanuj |
| author_facet | Stoltz, Dustin S. Taylor, Marshall A. Kumar, Sanuj |
| contents | Currently, there are thousands of large pretrained language models (LLMs) available to social scientists. How do we select among them? Using validity, reliability, reproducibility, and replicability as guides, we explore the significance of: (1) model openness, (2) model footprint, (3) training data, and (4) model architectures and fine-tuning. While ex-ante tests of validity (i.e., benchmarks) are often privileged in these discussions, we argue that social scientists cannot altogether avoid validating computational measures (ex-post). Replicability, in particular, is a more pressing guide for selecting language models. Being able to reliably replicate a particular finding that entails the use of a language model necessitates reliably reproducing a task. To this end, we propose starting with smaller, open models, and constructing delimited benchmarks to demonstrate the validity of the entire computational pipeline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_10926 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Selecting Language Models for Social Science: Start Small, Start Open, and Validate Stoltz, Dustin S. Taylor, Marshall A. Kumar, Sanuj Computation and Language Artificial Intelligence Currently, there are thousands of large pretrained language models (LLMs) available to social scientists. How do we select among them? Using validity, reliability, reproducibility, and replicability as guides, we explore the significance of: (1) model openness, (2) model footprint, (3) training data, and (4) model architectures and fine-tuning. While ex-ante tests of validity (i.e., benchmarks) are often privileged in these discussions, we argue that social scientists cannot altogether avoid validating computational measures (ex-post). Replicability, in particular, is a more pressing guide for selecting language models. Being able to reliably replicate a particular finding that entails the use of a language model necessitates reliably reproducing a task. To this end, we propose starting with smaller, open models, and constructing delimited benchmarks to demonstrate the validity of the entire computational pipeline. |
| title | Selecting Language Models for Social Science: Start Small, Start Open, and Validate |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.10926 |