Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.15056 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911216283156480 |
|---|---|
| author | Lu, Ziqing Hassibi, Babak Lai, Lifeng Xu, Weiyu |
| author_facet | Lu, Ziqing Hassibi, Babak Lai, Lifeng Xu, Weiyu |
| contents | Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_15056 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions Lu, Ziqing Hassibi, Babak Lai, Lifeng Xu, Weiyu Machine Learning Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward. |
| title | Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.15056 |