Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929684912013312 |
|---|---|
| author | Gao, Yuan Lee, Dokyun Burtch, Gordon Fazelpour, Sina |
| author_facet | Gao, Yuan Lee, Dokyun Burtch, Gordon Fazelpour, Sina |
| contents | Recent studies suggest large language models (LLMs) can exhibit human-like reasoning, aligning with human behavior in economic experiments, surveys, and political discourse. This has led many to propose that LLMs can be used as surrogates or simulations for humans in social science research. However, LLMs differ fundamentally from humans, relying on probabilistic patterns, absent the embodied experiences or survival objectives that shape human cognition. We assess the reasoning depth of LLMs using the 11-20 money request game. Nearly all advanced approaches fail to replicate human behavior distributions across many models. Causes of failure are diverse and unpredictable, relating to input language, roles, and safeguarding. These results advise caution when using LLMs to study human behavior or as surrogates or simulations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19599 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina Gao, Yuan Lee, Dokyun Burtch, Gordon Fazelpour, Sina General Economics Economics Artificial Intelligence Computers and Society Human-Computer Interaction Recent studies suggest large language models (LLMs) can exhibit human-like reasoning, aligning with human behavior in economic experiments, surveys, and political discourse. This has led many to propose that LLMs can be used as surrogates or simulations for humans in social science research. However, LLMs differ fundamentally from humans, relying on probabilistic patterns, absent the embodied experiences or survival objectives that shape human cognition. We assess the reasoning depth of LLMs using the 11-20 money request game. Nearly all advanced approaches fail to replicate human behavior distributions across many models. Causes of failure are diverse and unpredictable, relating to input language, roles, and safeguarding. These results advise caution when using LLMs to study human behavior or as surrogates or simulations. |
| title | Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina |
| topic | General Economics Economics Artificial Intelligence Computers and Society Human-Computer Interaction |
| url | https://arxiv.org/abs/2410.19599 |