Exploring Social Desirability Response Bias in Large Language Models: Evidence from GPT-4 Simulations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909357035225088 |
|---|---|
| author | Lee, Sanguk Yang, Kai-Qi Peng, Tai-Quan Heo, Ruth Liu, Hui |
| author_facet | Lee, Sanguk Yang, Kai-Qi Peng, Tai-Quan Heo, Ruth Liu, Hui |
| contents | Large language models (LLMs) are employed to simulate human-like responses in social surveys, yet it remains unclear if they develop biases like social desirability response (SDR) bias. To investigate this, GPT-4 was assigned personas from four societies, using data from the 2022 Gallup World Poll. These synthetic samples were then prompted with or without a commitment statement intended to induce SDR. The results were mixed. While the commitment statement increased SDR index scores, suggesting SDR bias, it reduced civic engagement scores, indicating an opposite trend. Additional findings revealed demographic associations with SDR scores and showed that the commitment statement had limited impact on GPT-4's predictive performance. The study underscores potential avenues for using LLMs to investigate biases in both humans and LLMs themselves. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_15442 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring Social Desirability Response Bias in Large Language Models: Evidence from GPT-4 Simulations Lee, Sanguk Yang, Kai-Qi Peng, Tai-Quan Heo, Ruth Liu, Hui Artificial Intelligence Computers and Society Large language models (LLMs) are employed to simulate human-like responses in social surveys, yet it remains unclear if they develop biases like social desirability response (SDR) bias. To investigate this, GPT-4 was assigned personas from four societies, using data from the 2022 Gallup World Poll. These synthetic samples were then prompted with or without a commitment statement intended to induce SDR. The results were mixed. While the commitment statement increased SDR index scores, suggesting SDR bias, it reduced civic engagement scores, indicating an opposite trend. Additional findings revealed demographic associations with SDR scores and showed that the commitment statement had limited impact on GPT-4's predictive performance. The study underscores potential avenues for using LLMs to investigate biases in both humans and LLMs themselves. |
| title | Exploring Social Desirability Response Bias in Large Language Models: Evidence from GPT-4 Simulations |
| topic | Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2410.15442 |