Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.06261 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916782504148992 |
|---|---|
| author | Jeong, Jihwan Wang, Xiaoyu Wang, Jingmin Sanner, Scott Poupart, Pascal |
| author_facet | Jeong, Jihwan Wang, Xiaoyu Wang, Jingmin Sanner, Scott Poupart, Pascal |
| contents | Offline reinforcement learning (RL) is crucial when online exploration is costly or unsafe but often struggles with high epistemic uncertainty due to limited data. Existing methods rely on fixed conservative policies, restricting adaptivity and generalization. To address this, we propose Reflect-then-Plan (RefPlan), a novel doubly Bayesian offline model-based (MB) planning approach. RefPlan unifies uncertainty modeling and MB planning by recasting planning as Bayesian posterior estimation. At deployment, it updates a belief over environment dynamics using real-time observations, incorporating uncertainty into MB planning via marginalization. Empirical results on standard benchmarks show that RefPlan significantly improves the performance of conservative offline RL policies. In particular, RefPlan maintains robust performance under high epistemic uncertainty and limited data, while demonstrating resilience to changing environment dynamics, improving the flexibility, generalizability, and robustness of offline-learned policies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06261 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens Jeong, Jihwan Wang, Xiaoyu Wang, Jingmin Sanner, Scott Poupart, Pascal Artificial Intelligence Machine Learning Offline reinforcement learning (RL) is crucial when online exploration is costly or unsafe but often struggles with high epistemic uncertainty due to limited data. Existing methods rely on fixed conservative policies, restricting adaptivity and generalization. To address this, we propose Reflect-then-Plan (RefPlan), a novel doubly Bayesian offline model-based (MB) planning approach. RefPlan unifies uncertainty modeling and MB planning by recasting planning as Bayesian posterior estimation. At deployment, it updates a belief over environment dynamics using real-time observations, incorporating uncertainty into MB planning via marginalization. Empirical results on standard benchmarks show that RefPlan significantly improves the performance of conservative offline RL policies. In particular, RefPlan maintains robust performance under high epistemic uncertainty and limited data, while demonstrating resilience to changing environment dynamics, improving the flexibility, generalizability, and robustness of offline-learned policies. |
| title | Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2506.06261 |