Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917333589557248 |
|---|---|
| author | Zhou, Xuhui Sun, Weiwei Ma, Qianou Xie, Yiqing Liu, Jiarui Du, Weihua Welleck, Sean Yang, Yiming Neubig, Graham Wu, Sherry Tongshuang Sap, Maarten |
| author_facet | Zhou, Xuhui Sun, Weiwei Ma, Qianou Xie, Yiqing Liu, Jiarui Du, Weihua Welleck, Sean Yang, Yiming Neubig, Graham Wu, Sherry Tongshuang Sap, Maarten |
| contents | As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full $τ$-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_11245 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Mind the Sim2Real Gap in User Simulation for Agentic Tasks Zhou, Xuhui Sun, Weiwei Ma, Qianou Xie, Yiqing Liu, Jiarui Du, Weihua Welleck, Sean Yang, Yiming Neubig, Graham Wu, Sherry Tongshuang Sap, Maarten Artificial Intelligence As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full $τ$-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation. |
| title | Mind the Sim2Real Gap in User Simulation for Agentic Tasks |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.11245 |