UserRL: Training Interactive User-Centric Agent via Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918147129344000 |
|---|---|
| author | Qian, Cheng Liu, Zuxin Prabhakar, Akshara Qiu, Jielin Liu, Zhiwei Chen, Haolin Kokane, Shirley Ji, Heng Yao, Weiran Heinecke, Shelby Savarese, Silvio Xiong, Caiming Wang, Huan |
| author_facet | Qian, Cheng Liu, Zuxin Prabhakar, Akshara Qiu, Jielin Liu, Zhiwei Chen, Haolin Kokane, Shirley Ji, Heng Yao, Weiran Heinecke, Shelby Savarese, Silvio Xiong, Caiming Wang, Huan |
| contents | Reinforcement learning (RL) has shown promise in training agentic models that move beyond static benchmarks to engage in dynamic, multi-turn interactions. Yet, the ultimate value of such agents lies in their ability to assist users, a setting where diversity and dynamics of user interaction pose challenges. In this work, we propose UserRL, a unified framework for training and evaluating user-centric abilities through standardized gym environments paired with simulated users. We systematically vary turn-level reward assignment and trajectory-level score calculation to analyze how different formulations affect learning under the GRPO algorithm. Our experiments across Qwen3 models reveal three key findings: (i) SFT cold start is critical for unlocking initial interaction ability and enabling sustained RL improvements; (ii) deliberate trajectory scoring yields more efficient and effective multi-turn interactions; and (iii) while stronger simulated users (e.g., GPT-4o) facilitates training, open-source simulators (e.g., Qwen3-32B) remain a cost-effective and transferable option. Together, these results highlight that careful design of reward shaping and user simulation choice is as crucial as model scale, and establish UserRL as a practical pathway for developing robust user-centric agentic models. All codes and data are public for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19736 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UserRL: Training Interactive User-Centric Agent via Reinforcement Learning Qian, Cheng Liu, Zuxin Prabhakar, Akshara Qiu, Jielin Liu, Zhiwei Chen, Haolin Kokane, Shirley Ji, Heng Yao, Weiran Heinecke, Shelby Savarese, Silvio Xiong, Caiming Wang, Huan Artificial Intelligence Computation and Language Machine Learning Reinforcement learning (RL) has shown promise in training agentic models that move beyond static benchmarks to engage in dynamic, multi-turn interactions. Yet, the ultimate value of such agents lies in their ability to assist users, a setting where diversity and dynamics of user interaction pose challenges. In this work, we propose UserRL, a unified framework for training and evaluating user-centric abilities through standardized gym environments paired with simulated users. We systematically vary turn-level reward assignment and trajectory-level score calculation to analyze how different formulations affect learning under the GRPO algorithm. Our experiments across Qwen3 models reveal three key findings: (i) SFT cold start is critical for unlocking initial interaction ability and enabling sustained RL improvements; (ii) deliberate trajectory scoring yields more efficient and effective multi-turn interactions; and (iii) while stronger simulated users (e.g., GPT-4o) facilitates training, open-source simulators (e.g., Qwen3-32B) remain a cost-effective and transferable option. Together, these results highlight that careful design of reward shaping and user simulation choice is as crucial as model scale, and establish UserRL as a practical pathway for developing robust user-centric agentic models. All codes and data are public for future research. |
| title | UserRL: Training Interactive User-Centric Agent via Reinforcement Learning |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2509.19736 |