Policy and World Modeling Co-Training for Language Agents
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916075001610240 |
|---|---|
| author | Lu, Ning Lin, Baijiong Liu, Shengcai Wu, Jiahao Lv, Haoze Wei, Yanbin Zhu, Lingting Qian, Shengju Wang, Xin Chen, Ying-Cong Wang, Qi Tang, Ke |
| author_facet | Lu, Ning Lin, Baijiong Liu, Shengcai Wu, Jiahao Lv, Haoze Wei, Yanbin Zhu, Lingting Qian, Shengju Wang, Xin Chen, Ying-Cong Wang, Qi Tang, Ke |
| contents | Reinforcement learning (RL) improves large language model (LLM) agents by teaching them which actions lead to high rewards, but provides little supervision on what those actions do to the environment. World modeling (WM) can fill this gap, yet existing approaches often require separate simulators, extra training stages, or additional inference-time computation. We observe that on-policy RL rollouts already contain the needed signal: each transition pairs an action with its resulting next observation. Based on this observation, we propose PaW, a Policy and World modeling co-training framework that adds auxiliary WM supervision to the same policy during RL, without changing the inference paradigm. To make auxiliary WM supervision informative and stable, PaW introduces three components: action-entropy-based WM data selection, noise-tolerant WM loss, and reward-adaptive loss balancing. Experiments on three agentic task benchmarks show consistent improvements over strong RL baselines across models and RL algorithms. These results suggest that standard RL rollouts are a practical source of WM supervision for language-agent training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_02388 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Policy and World Modeling Co-Training for Language Agents Lu, Ning Lin, Baijiong Liu, Shengcai Wu, Jiahao Lv, Haoze Wei, Yanbin Zhu, Lingting Qian, Shengju Wang, Xin Chen, Ying-Cong Wang, Qi Tang, Ke Machine Learning Artificial Intelligence Reinforcement learning (RL) improves large language model (LLM) agents by teaching them which actions lead to high rewards, but provides little supervision on what those actions do to the environment. World modeling (WM) can fill this gap, yet existing approaches often require separate simulators, extra training stages, or additional inference-time computation. We observe that on-policy RL rollouts already contain the needed signal: each transition pairs an action with its resulting next observation. Based on this observation, we propose PaW, a Policy and World modeling co-training framework that adds auxiliary WM supervision to the same policy during RL, without changing the inference paradigm. To make auxiliary WM supervision informative and stable, PaW introduces three components: action-entropy-based WM data selection, noise-tolerant WM loss, and reward-adaptive loss balancing. Experiments on three agentic task benchmarks show consistent improvements over strong RL baselines across models and RL algorithms. These results suggest that standard RL rollouts are a practical source of WM supervision for language-agent training. |
| title | Policy and World Modeling Co-Training for Language Agents |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2606.02388 |