The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918430152589312 |
|---|---|
| author | Zhou, Jiaxu Huang, Jen-tse Zhou, Xuhui Lam, Man Ho Wang, Xintao Zhu, Hao Wang, Wenxuan Sap, Maarten |
| author_facet | Zhou, Jiaxu Huang, Jen-tse Zhou, Xuhui Lam, Man Ho Wang, Xintao Zhu, Hao Wang, Wenxuan Sap, Maarten |
| contents | Large language models (LLMs) are increasingly deployed to simulate human collective behaviors, yet the methodological rigor of these "AI societies" remains under-explored. Through a systematic audit of 39 recent studies, we identify six pervasive flaws-spanning agent profiles, interaction, memory, control, unawareness, and realism (PIMMUR). Our analysis reveals that 89.7% of studies violate at least one principle, undermining simulation validity. We demonstrate that frontier LLMs correctly identify the underlying social experiment in 50.8% of cases, while 61.0% of prompts exert excessive control that pre-determines outcomes. By reproducing five representative experiments (e.g., telephone game), we show that reported collective phenomena often vanish or reverse when PIMMUR principles are enforced, suggesting that many "emergent" behaviors are methodological artifacts rather than genuine social dynamics. Our findings suggest that current AI simulations may capture model-specific biases rather than universal human social behaviors, raising critical concerns about the use of LLMs as scientific proxies for human society. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_18052 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies Zhou, Jiaxu Huang, Jen-tse Zhou, Xuhui Lam, Man Ho Wang, Xintao Zhu, Hao Wang, Wenxuan Sap, Maarten Computation and Language Computers and Society Large language models (LLMs) are increasingly deployed to simulate human collective behaviors, yet the methodological rigor of these "AI societies" remains under-explored. Through a systematic audit of 39 recent studies, we identify six pervasive flaws-spanning agent profiles, interaction, memory, control, unawareness, and realism (PIMMUR). Our analysis reveals that 89.7% of studies violate at least one principle, undermining simulation validity. We demonstrate that frontier LLMs correctly identify the underlying social experiment in 50.8% of cases, while 61.0% of prompts exert excessive control that pre-determines outcomes. By reproducing five representative experiments (e.g., telephone game), we show that reported collective phenomena often vanish or reverse when PIMMUR principles are enforced, suggesting that many "emergent" behaviors are methodological artifacts rather than genuine social dynamics. Our findings suggest that current AI simulations may capture model-specific biases rather than universal human social behaviors, raising critical concerns about the use of LLMs as scientific proxies for human society. |
| title | The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies |
| topic | Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2509.18052 |