Constrained Average-Reward Intermittently Observable MDPs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914026874732544 |
|---|---|
| author | Avrachenkov, Konstantin Dhiman, Madhu Kavitha, Veeraruna |
| author_facet | Avrachenkov, Konstantin Dhiman, Madhu Kavitha, Veeraruna |
| contents | In Markov Decision Processes (MDPs) with intermittent state information, decision-making becomes challenging due to periods of missing observations. Linear programming (LP) methods can play a crucial role in solving MDPs, in particular, with constraints. However, the resultant belief MDPs lead to infinite dimensional LPs, even when the original MDP is with finite state and action spaces. The verification of strong duality becomes non-trivial. This paper investigates the conditions for no duality gap in average-reward finite Markov decision process with intermittent state observations. We first establish that in such MDPs, the belief MDP is unichain if the original Markov chain is recurrent. Furthermore, we establish strong duality of the problem under the same assumption. Finally, we provide a wireless channel example, where the belief state depends on the last channel state received and the age of the channel state. Our numerical results indicate interesting properties of the solution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_13823 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Constrained Average-Reward Intermittently Observable MDPs Avrachenkov, Konstantin Dhiman, Madhu Kavitha, Veeraruna Optimization and Control In Markov Decision Processes (MDPs) with intermittent state information, decision-making becomes challenging due to periods of missing observations. Linear programming (LP) methods can play a crucial role in solving MDPs, in particular, with constraints. However, the resultant belief MDPs lead to infinite dimensional LPs, even when the original MDP is with finite state and action spaces. The verification of strong duality becomes non-trivial. This paper investigates the conditions for no duality gap in average-reward finite Markov decision process with intermittent state observations. We first establish that in such MDPs, the belief MDP is unichain if the original Markov chain is recurrent. Furthermore, we establish strong duality of the problem under the same assumption. Finally, we provide a wireless channel example, where the belief state depends on the last channel state received and the age of the channel state. Our numerical results indicate interesting properties of the solution. |
| title | Constrained Average-Reward Intermittently Observable MDPs |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2504.13823 |