Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.08260 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912332198707200 |
|---|---|
| author | Khaokaew, Yonchanok Salim, Flora D. Züfle, Andreas Xue, Hao Anderson, Taylor MacIntyre, C. Raina Scotch, Matthew Heslop, David J |
| author_facet | Khaokaew, Yonchanok Salim, Flora D. Züfle, Andreas Xue, Hao Anderson, Taylor MacIntyre, C. Raina Scotch, Matthew Heslop, David J |
| contents | Generative agents have been increasingly used to simulate human behaviour in silico, driven by large language models (LLMs). These simulacra serve as sandboxes for studying human behaviour without compromising privacy or safety. However, it remains unclear whether such agents can truly represent real individuals. This work compares survey data from the Understanding America Study (UAS) on healthcare decision-making with simulated responses from generative agents. Using demographic-based prompt engineering, we create digital twins of survey respondents and analyse how well different LLMs reproduce real-world behaviours. Our findings show that some LLMs fail to reflect realistic decision-making, such as predicting universal vaccine acceptance. However, Llama 3 captures variations across race and Income more accurately but also introduces biases not present in the UAS data. This study highlights the potential of generative agents for behavioural research while underscoring the risks of bias from both LLMs and prompting strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08260 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating the Bias in LLMs for Surveying Opinion and Decision Making in Healthcare Khaokaew, Yonchanok Salim, Flora D. Züfle, Andreas Xue, Hao Anderson, Taylor MacIntyre, C. Raina Scotch, Matthew Heslop, David J Computation and Language Generative agents have been increasingly used to simulate human behaviour in silico, driven by large language models (LLMs). These simulacra serve as sandboxes for studying human behaviour without compromising privacy or safety. However, it remains unclear whether such agents can truly represent real individuals. This work compares survey data from the Understanding America Study (UAS) on healthcare decision-making with simulated responses from generative agents. Using demographic-based prompt engineering, we create digital twins of survey respondents and analyse how well different LLMs reproduce real-world behaviours. Our findings show that some LLMs fail to reflect realistic decision-making, such as predicting universal vaccine acceptance. However, Llama 3 captures variations across race and Income more accurately but also introduces biases not present in the UAS data. This study highlights the potential of generative agents for behavioural research while underscoring the risks of bias from both LLMs and prompting strategies. |
| title | Evaluating the Bias in LLMs for Surveying Opinion and Decision Making in Healthcare |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.08260 |