Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.13824 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911595586650112 |
|---|---|
| author | Huang, Youling Chen, Guanqiao Yao, Junchi Wang, Lu Yang, Fangkai Du, Chao Zhao, ChenZhuo Zhao, Pu Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei |
| author_facet | Huang, Youling Chen, Guanqiao Yao, Junchi Wang, Lu Yang, Fangkai Du, Chao Zhao, ChenZhuo Zhao, Pu Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei |
| contents | World models have been emerging as critical components for assessing the consequences of actions generated by interactive agents in online planning and offline evaluation. In text-based environments, world models are typically evaluated and trained with single-step metrics such as Exact Match, aiming to improve the similarity between predicted and real-world states, but such metrics have been shown to be insufficient for capturing actual agent behavior. To address this issue, we introduce a new behavior-aligned training paradigm aimed at improving the functional consistency between the world model and the real environment. This paradigm focuses on optimizing a tractable step-level metric named Behavior Consistency Reward (BehR), which measures how much the likelihood of a logged next action changes between the real state and the world-model-predicted state under a frozen Reference Agent. Experiments on WebShop and TextWorld show that BehR-based training improves long-term alignment in several settings, with the clearest gains in WebShop and less movement in near-ceiling regimes, while preserving or improving single-step prediction quality in three of four settings. World models trained with BehR also achieve lower false positives in offline surrogate evaluation and show modest but encouraging gains in inference-time lookahead planning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_13824 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond State Consistency: Behavior Consistency in Text-Based World Models Huang, Youling Chen, Guanqiao Yao, Junchi Wang, Lu Yang, Fangkai Du, Chao Zhao, ChenZhuo Zhao, Pu Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Machine Learning World models have been emerging as critical components for assessing the consequences of actions generated by interactive agents in online planning and offline evaluation. In text-based environments, world models are typically evaluated and trained with single-step metrics such as Exact Match, aiming to improve the similarity between predicted and real-world states, but such metrics have been shown to be insufficient for capturing actual agent behavior. To address this issue, we introduce a new behavior-aligned training paradigm aimed at improving the functional consistency between the world model and the real environment. This paradigm focuses on optimizing a tractable step-level metric named Behavior Consistency Reward (BehR), which measures how much the likelihood of a logged next action changes between the real state and the world-model-predicted state under a frozen Reference Agent. Experiments on WebShop and TextWorld show that BehR-based training improves long-term alignment in several settings, with the clearest gains in WebShop and less movement in near-ceiling regimes, while preserving or improving single-step prediction quality in three of four settings. World models trained with BehR also achieve lower false positives in offline surrogate evaluation and show modest but encouraging gains in inference-time lookahead planning. |
| title | Beyond State Consistency: Behavior Consistency in Text-Based World Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2604.13824 |