Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.18462 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914341468504064 |
|---|---|
| author | Morocho, Erika Elizabeth Taday Cima, Lorenzo Fagni, Tiziano Avvenuti, Marco Cresci, Stefano |
| author_facet | Morocho, Erika Elizabeth Taday Cima, Lorenzo Fagni, Tiziano Avvenuti, Marco Cresci, Stefano |
| contents | Using persona-conditioned LLMs as synthetic survey respondents has become a common practice in computational social science and agent-based simulations. Yet, it remains unclear whether multi-attribute persona prompting improves LLM reliability or instead introduces distortions. Here we contribute to this assessment by leveraging a large dataset of U.S. microdata from the World Values Survey. Concretely, we evaluate two open-weight chat models and a random-guesser baseline across more than 70K respondent-item instances. We find that persona prompting does not yield a clear aggregate improvement in survey alignment and, in many cases, significantly degrades performance. Persona effects are highly heterogeneous as most items exhibit minimal change, while a small subset of questions and underrepresented subgroups experience disproportionate distortions. Our findings highlight a key adverse impact of current persona-based simulation practices: demographic conditioning can redistribute error in ways that undermine subgroup fidelity and risk misleading downstream analyses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_18462 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondents Morocho, Erika Elizabeth Taday Cima, Lorenzo Fagni, Tiziano Avvenuti, Marco Cresci, Stefano Computers and Society Artificial Intelligence Using persona-conditioned LLMs as synthetic survey respondents has become a common practice in computational social science and agent-based simulations. Yet, it remains unclear whether multi-attribute persona prompting improves LLM reliability or instead introduces distortions. Here we contribute to this assessment by leveraging a large dataset of U.S. microdata from the World Values Survey. Concretely, we evaluate two open-weight chat models and a random-guesser baseline across more than 70K respondent-item instances. We find that persona prompting does not yield a clear aggregate improvement in survey alignment and, in many cases, significantly degrades performance. Persona effects are highly heterogeneous as most items exhibit minimal change, while a small subset of questions and underrepresented subgroups experience disproportionate distortions. Our findings highlight a key adverse impact of current persona-based simulation practices: demographic conditioning can redistribute error in ways that undermine subgroup fidelity and risk misleading downstream analyses. |
| title | Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondents |
| topic | Computers and Society Artificial Intelligence |
| url | https://arxiv.org/abs/2602.18462 |