UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917203357466624 |
|---|---|
| author | Zhang, Feng Li, Shijia Zhang, Chunmao Ma, Zhanyu Xu, Jun Gao, Jiuchong Hao, Jinghua He, Renqing Xu, Jingwen Liu, Han |
| author_facet | Zhang, Feng Li, Shijia Zhang, Chunmao Ma, Zhanyu Xu, Jun Gao, Jiuchong Hao, Jinghua He, Renqing Xu, Jingwen Liu, Han |
| contents | User simulators serve as the critical interactive environment for agent post-training, and an ideal user simulator generalizes across domains and proactively engages in negotiation by challenging or bargaining. However, current methods exhibit two issues. They rely on static and context-unaware profiles, necessitating extensive manual redesign for new scenarios, thus limiting generalizability. Moreover, they neglect human strategic thinking, leading to vulnerability to agent manipulation. To address these issues, we propose UserLM-R1, a novel user language model with reasoning capability. Specifically, we first construct comprehensive user profiles with both static roles and dynamic scenario-specific goals for adaptation to diverse scenarios. Then, we propose a goal-driven decision-making policy to generate high-quality rationales before producing responses, and further refine the reasoning and improve strategic capabilities with supervised fine-tuning and multi-reward reinforcement learning. Extensive experimental results demonstrate that UserLM-R1 outperforms competitive baselines, particularly on the more challenging adversarial set. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_09215 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning Zhang, Feng Li, Shijia Zhang, Chunmao Ma, Zhanyu Xu, Jun Gao, Jiuchong Hao, Jinghua He, Renqing Xu, Jingwen Liu, Han Computation and Language User simulators serve as the critical interactive environment for agent post-training, and an ideal user simulator generalizes across domains and proactively engages in negotiation by challenging or bargaining. However, current methods exhibit two issues. They rely on static and context-unaware profiles, necessitating extensive manual redesign for new scenarios, thus limiting generalizability. Moreover, they neglect human strategic thinking, leading to vulnerability to agent manipulation. To address these issues, we propose UserLM-R1, a novel user language model with reasoning capability. Specifically, we first construct comprehensive user profiles with both static roles and dynamic scenario-specific goals for adaptation to diverse scenarios. Then, we propose a goal-driven decision-making policy to generate high-quality rationales before producing responses, and further refine the reasoning and improve strategic capabilities with supervised fine-tuning and multi-reward reinforcement learning. Extensive experimental results demonstrate that UserLM-R1 outperforms competitive baselines, particularly on the more challenging adversarial set. |
| title | UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.09215 |