Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914322323603456 |
|---|---|
| author | Zorba, Denis Šiška, David Szpruch, Lukasz |
| author_facet | Zorba, Denis Šiška, David Szpruch, Lukasz |
| contents | We provide theoretical guarantees for convergence of discrete-time policy mirror descent with inexact advantage functions updated using temporal difference (TD) learning for entropy regularised MDPs in Polish state and action spaces. We rigorously derive sufficient conditions under which the single-loop actor-critic scheme is stable and convergent. To weaken these conditions, we introduce a variant that performs multiple TD steps per policy update and derive an explicit lower bound on the number of TD steps required to ensure stability. Finally, we establish sub-linear convergence when the number of TD steps grows logarithmically with the number of policy updates, and linear convergence when it grows linearly under a concentrability assumption. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_10838 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence Zorba, Denis Šiška, David Szpruch, Lukasz Optimization and Control We provide theoretical guarantees for convergence of discrete-time policy mirror descent with inexact advantage functions updated using temporal difference (TD) learning for entropy regularised MDPs in Polish state and action spaces. We rigorously derive sufficient conditions under which the single-loop actor-critic scheme is stable and convergent. To weaken these conditions, we introduce a variant that performs multiple TD steps per policy update and derive an explicit lower bound on the number of TD steps required to ensure stability. Finally, we establish sub-linear convergence when the number of TD steps grows logarithmically with the number of policy updates, and linear convergence when it grows linearly under a concentrability assumption. |
| title | Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2602.10838 |