Validated Hypotheses as a Lens for Human-Likeness Evaluation in AI Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914568740012032 |
|---|---|
| author | Liu, Xuan Shang, HaoYang Liu, Zizhang Feng, Yuanjun Zhai, Guankai Xiao, Yunze Tu, Yiwen Jin, Haojian |
| author_facet | Liu, Xuan Shang, HaoYang Liu, Zizhang Feng, Yuanjun Zhai, Guankai Xiao, Yunze Tu, Yiwen Jin, Haojian |
| contents | We propose using validated behavioral hypotheses as a lens for evaluating human-likeness in LLM-based agents. Our key idea is simple: If an agent is human-like, a population of such agents should reach the same inferential conclusion as the human population when run through the same experiment. Decades of social science have produced many such validated findings, each anchored to concrete experimental protocols and robustly established through independent replication. This yields an evaluation that is objective, decomposable, and scalable. We operationalize this lens through HumanStudy-Bench, an open platform that turns published human-subject studies into reusable simulation environments and administers the evaluation to configurable agents. It scores agent-human alignment on two metrics: the Probability Alignment Score (PAS) for inferential agreement and the Effect Consistency Score (ECS) for effect-size agreement. We curated an initial suite of 12 studies whose hypotheses are robustly established through independent replication, and evaluated 10 models under 4 agent designs. Results show that agent responses polarize between full replication and complete failure; agent design influences alignment more than model scale, but its effect is non-monotonic. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_15473 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Validated Hypotheses as a Lens for Human-Likeness Evaluation in AI Agents Liu, Xuan Shang, HaoYang Liu, Zizhang Feng, Yuanjun Zhai, Guankai Xiao, Yunze Tu, Yiwen Jin, Haojian Computers and Society We propose using validated behavioral hypotheses as a lens for evaluating human-likeness in LLM-based agents. Our key idea is simple: If an agent is human-like, a population of such agents should reach the same inferential conclusion as the human population when run through the same experiment. Decades of social science have produced many such validated findings, each anchored to concrete experimental protocols and robustly established through independent replication. This yields an evaluation that is objective, decomposable, and scalable. We operationalize this lens through HumanStudy-Bench, an open platform that turns published human-subject studies into reusable simulation environments and administers the evaluation to configurable agents. It scores agent-human alignment on two metrics: the Probability Alignment Score (PAS) for inferential agreement and the Effect Consistency Score (ECS) for effect-size agreement. We curated an initial suite of 12 studies whose hypotheses are robustly established through independent replication, and evaluated 10 models under 4 agent designs. Results show that agent responses polarize between full replication and complete failure; agent design influences alignment more than model scale, but its effect is non-monotonic. |
| title | Validated Hypotheses as a Lens for Human-Likeness Evaluation in AI Agents |
| topic | Computers and Society |
| url | https://arxiv.org/abs/2605.15473 |