Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910199154999296 |
|---|---|
| author | Zheng, Yaowei Zhang, Richong Wu, Shenxi Bian, Shirui Zhang, Haosong Zeng, Li Ma, Xingjian Zhang, Yichi |
| author_facet | Zheng, Yaowei Zhang, Richong Wu, Shenxi Bian, Shirui Zhang, Haosong Zeng, Li Ma, Xingjian Zhang, Yichi |
| contents | We study finite-horizon continuous-time policy evaluation from discrete closed-loop trajectories under time-inhomogeneous dynamics. The target value surface solves a backward parabolic equation, but the Bellman baseline obtained from one-step recursion is only first-order in the grid width. We estimate the time-dependent generator from multi-step transitions using moment-matching coefficients that cancel lower-order truncation terms, and combine the resulting surrogate with backward regression. The main theory gives an end-to-end decomposition into generator misspecification, projection error, pooling bias, finite-sample error, and start-up error, together with a decision-frequency regime map explaining when higher-order gains should be visible. Across calibration studies, four-scale benchmarks, feature and start-up ablations, and gain-mismatch stress tests, the second-order estimator consistently improves on the Bellman baseline and remains stable in the regime where the theory predicts visible gains. These results position high-order generator regression as an interpretable continuous-time policy-evaluation method with a clear operating region. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_18972 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation Zheng, Yaowei Zhang, Richong Wu, Shenxi Bian, Shirui Zhang, Haosong Zeng, Li Ma, Xingjian Zhang, Yichi Machine Learning Optimization and Control 93E35, 93E20, 49L20, 90C40 We study finite-horizon continuous-time policy evaluation from discrete closed-loop trajectories under time-inhomogeneous dynamics. The target value surface solves a backward parabolic equation, but the Bellman baseline obtained from one-step recursion is only first-order in the grid width. We estimate the time-dependent generator from multi-step transitions using moment-matching coefficients that cancel lower-order truncation terms, and combine the resulting surrogate with backward regression. The main theory gives an end-to-end decomposition into generator misspecification, projection error, pooling bias, finite-sample error, and start-up error, together with a decision-frequency regime map explaining when higher-order gains should be visible. Across calibration studies, four-scale benchmarks, feature and start-up ablations, and gain-mismatch stress tests, the second-order estimator consistently improves on the Bellman baseline and remains stable in the regime where the theory predicts visible gains. These results position high-order generator regression as an interpretable continuous-time policy-evaluation method with a clear operating region. |
| title | Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation |
| topic | Machine Learning Optimization and Control 93E35, 93E20, 49L20, 90C40 |
| url | https://arxiv.org/abs/2604.18972 |