Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.02812 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909768079114240 |
|---|---|
| author | He, Zixuan Charalambous, Charalambos D. Stavrou, Photios A. |
| author_facet | He, Zixuan Charalambous, Charalambos D. Stavrou, Photios A. |
| contents | This paper studies a finite-horizon Markov decision problem with information-theoretic constraints, where the goal is to minimize directed information from the controlled source process to the control process, subject to stage-wise cost constraints, aiming for an optimal control policy. We propose a new way of approximating a solution for this problem, which is known to be formulated as an unconstrained MDP with a continuous information-state using Q-factors. To avoid the computational complexity of discretizing the continuous information-state space, we propose a truncated rollout-based backward-forward approximate dynamic programming (ADP) framework. Our approach consists of two phases: an offline base policy approximation over a shorter time horizon, followed by an online rollout lookahead minimization, both supported by provable convergence guarantees. We supplement our theoretical results with a numerical example where we demonstrate the cost improvement of the rollout method compared to a previously proposed policy approximation method, and the computational complexity observed in executing the offline and online phases for the two methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_02812 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rollout-Based Approximate Dynamic Programming for MDPs with Information-Theoretic Constraints He, Zixuan Charalambous, Charalambos D. Stavrou, Photios A. Systems and Control Information Theory This paper studies a finite-horizon Markov decision problem with information-theoretic constraints, where the goal is to minimize directed information from the controlled source process to the control process, subject to stage-wise cost constraints, aiming for an optimal control policy. We propose a new way of approximating a solution for this problem, which is known to be formulated as an unconstrained MDP with a continuous information-state using Q-factors. To avoid the computational complexity of discretizing the continuous information-state space, we propose a truncated rollout-based backward-forward approximate dynamic programming (ADP) framework. Our approach consists of two phases: an offline base policy approximation over a shorter time horizon, followed by an online rollout lookahead minimization, both supported by provable convergence guarantees. We supplement our theoretical results with a numerical example where we demonstrate the cost improvement of the rollout method compared to a previously proposed policy approximation method, and the computational complexity observed in executing the offline and online phases for the two methods. |
| title | Rollout-Based Approximate Dynamic Programming for MDPs with Information-Theoretic Constraints |
| topic | Systems and Control Information Theory |
| url | https://arxiv.org/abs/2509.02812 |