PAC-Bayesian Reinforcement Learning Trains Generalizable Policies
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911729023188992 |
|---|---|
| author | Zitouni, Abdelkrim Hennequin, Mehdi Agoun, Juba Horache, Ryan Kabachi, Nadia Rivasplata, Omar |
| author_facet | Zitouni, Abdelkrim Hennequin, Mehdi Agoun, Juba Horache, Ryan Kabachi, Nadia Rivasplata, Omar |
| contents | We derive a novel PAC-Bayesian generalization bound for reinforcement learning that explicitly accounts for Markov dependencies in the data, through the chain's mixing time. This contributes to overcoming challenges in obtaining generalization guarantees for reinforcement learning, where the sequential nature of data breaks the independence assumptions underlying classical bounds. The new bound provides non-vacuous certificates for modern off-policy algorithms such as Soft Actor-Critic. We demonstrate the practical utility of the bound through PB-SAC, a novel algorithm that optimizes the bound during training to guide exploration. Experiments across several continuous control tasks show that the proposed approach provides meaningful confidence certificates while maintaining competitive performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10544 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PAC-Bayesian Reinforcement Learning Trains Generalizable Policies Zitouni, Abdelkrim Hennequin, Mehdi Agoun, Juba Horache, Ryan Kabachi, Nadia Rivasplata, Omar Machine Learning Artificial Intelligence We derive a novel PAC-Bayesian generalization bound for reinforcement learning that explicitly accounts for Markov dependencies in the data, through the chain's mixing time. This contributes to overcoming challenges in obtaining generalization guarantees for reinforcement learning, where the sequential nature of data breaks the independence assumptions underlying classical bounds. The new bound provides non-vacuous certificates for modern off-policy algorithms such as Soft Actor-Critic. We demonstrate the practical utility of the bound through PB-SAC, a novel algorithm that optimizes the bound during training to guide exploration. Experiments across several continuous control tasks show that the proposed approach provides meaningful confidence certificates while maintaining competitive performance. |
| title | PAC-Bayesian Reinforcement Learning Trains Generalizable Policies |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.10544 |