Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.06337 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911624532590592 |
|---|---|
| author | Zhao, Jianpeng Yuan, Chenyu Luo, Weiming Xie, Haoling Zhang, Guangwei Quan, Steven Jige Yuan, Zixuan Wang, Pengyang Zhang, Denghui |
| author_facet | Zhao, Jianpeng Yuan, Chenyu Luo, Weiming Xie, Haoling Zhang, Guangwei Quan, Steven Jige Yuan, Zixuan Wang, Pengyang Zhang, Denghui |
| contents | Questionnaire-based surveys are foundational to social science research and public policymaking, yet traditional survey methods remain costly, time-consuming, and often limited in scale. Although prior work has explored large language models (LLMs) as virtual survey respondents, existing studies often address narrow task settings, focus on single sociological domains, or lack a unified evaluation framework that enables systematic comparison across diverse datasets and models. To address these gaps, we introduce two complementary task abstractions: Partial Attribute Simulation (PAS), where LLMs predict missing attributes from incomplete respondent profiles, and Full Attribute Simulation (FAS), where LLMs generate complete synthetic datasets under zero-context and context-enhanced conditions. Both are framed as diagnostic and exploratory tools rather than replacements for human data collection. We curate LLM-S^3 (Large Language Model-based Sociodemographic Survey Simulation), a benchmark spanning 11 real-world public datasets across four sociological domains, and evaluate GPT-3.5/4 Turbo and LLaMA 3.0/3.1-8B under zero-shot and few-shot settings. Our evaluation reveals consistent performance trends across model families, highlights failure modes in structured output generation, and demonstrates how context and prompt design affect simulation fidelity. Our code and dataset are available at: https://github.com/dart-lab-research/LLM-S-Cube-Benchmark |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_06337 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation Zhao, Jianpeng Yuan, Chenyu Luo, Weiming Xie, Haoling Zhang, Guangwei Quan, Steven Jige Yuan, Zixuan Wang, Pengyang Zhang, Denghui Artificial Intelligence Questionnaire-based surveys are foundational to social science research and public policymaking, yet traditional survey methods remain costly, time-consuming, and often limited in scale. Although prior work has explored large language models (LLMs) as virtual survey respondents, existing studies often address narrow task settings, focus on single sociological domains, or lack a unified evaluation framework that enables systematic comparison across diverse datasets and models. To address these gaps, we introduce two complementary task abstractions: Partial Attribute Simulation (PAS), where LLMs predict missing attributes from incomplete respondent profiles, and Full Attribute Simulation (FAS), where LLMs generate complete synthetic datasets under zero-context and context-enhanced conditions. Both are framed as diagnostic and exploratory tools rather than replacements for human data collection. We curate LLM-S^3 (Large Language Model-based Sociodemographic Survey Simulation), a benchmark spanning 11 real-world public datasets across four sociological domains, and evaluate GPT-3.5/4 Turbo and LLaMA 3.0/3.1-8B under zero-shot and few-shot settings. Our evaluation reveals consistent performance trends across model families, highlights failure modes in structured output generation, and demonstrates how context and prompt design affect simulation fidelity. Our code and dataset are available at: https://github.com/dart-lab-research/LLM-S-Cube-Benchmark |
| title | Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.06337 |