LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918460198486016 |
|---|---|
| author | Park, Joon Sung Zou, Carolyn Q. Kamphorst, Jonne Egan, Niles Shaw, Aaron Hill, Benjamin Mako Cai, Carrie Morris, Meredith Ringel Liang, Percy Willer, Robb Bernstein, Michael S. |
| author_facet | Park, Joon Sung Zou, Carolyn Q. Kamphorst, Jonne Egan, Niles Shaw, Aaron Hill, Benjamin Mako Cai, Carrie Morris, Meredith Ringel Liang, Percy Willer, Robb Bernstein, Michael S. |
| contents | Machine learning can predict human behavior well when substantial structured data and well-defined outcomes are available, but these models are typically limited to specific outcomes and cannot readily be applied to new domains. We test whether large language models (LLMs) can support a more general-purpose approach by building person-specific simulations (i.e., "generative agents") grounded in self-report data. Using data from a diverse national sample of 1,052 Americans, we build agents from (i) two-hour, semi-structured interviews (elicited using the American Voices Project interview schedule), (ii) structured surveys (the General Social Survey and Big Five personality inventory), or (iii) both sources combined. On held-out General Social Survey items, agent accuracy reached 83% (interview only), 82% (surveys only), and 86% (combined) of participants' two-week test-retest consistency, compared with agents prompted only with individuals' demographics (74%). Agents predicted personality traits and behaviors in experiments with similar accuracy, and reduced disparities in accuracy across racial and ideological groups relative to demographics-only baselines. Together, these results show that LLMs agents grounded in rich qualitative or quantitative self-report data can support general-purpose simulation of individuals across outcomes, without requiring task-specific training data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_10109 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals Park, Joon Sung Zou, Carolyn Q. Kamphorst, Jonne Egan, Niles Shaw, Aaron Hill, Benjamin Mako Cai, Carrie Morris, Meredith Ringel Liang, Percy Willer, Robb Bernstein, Michael S. Artificial Intelligence Human-Computer Interaction Machine Learning Machine learning can predict human behavior well when substantial structured data and well-defined outcomes are available, but these models are typically limited to specific outcomes and cannot readily be applied to new domains. We test whether large language models (LLMs) can support a more general-purpose approach by building person-specific simulations (i.e., "generative agents") grounded in self-report data. Using data from a diverse national sample of 1,052 Americans, we build agents from (i) two-hour, semi-structured interviews (elicited using the American Voices Project interview schedule), (ii) structured surveys (the General Social Survey and Big Five personality inventory), or (iii) both sources combined. On held-out General Social Survey items, agent accuracy reached 83% (interview only), 82% (surveys only), and 86% (combined) of participants' two-week test-retest consistency, compared with agents prompted only with individuals' demographics (74%). Agents predicted personality traits and behaviors in experiments with similar accuracy, and reduced disparities in accuracy across racial and ideological groups relative to demographics-only baselines. Together, these results show that LLMs agents grounded in rich qualitative or quantitative self-report data can support general-purpose simulation of individuals across outcomes, without requiring task-specific training data. |
| title | LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals |
| topic | Artificial Intelligence Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2411.10109 |