Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theory
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915289286836224 |
|---|---|
| author | Zhang, Zhi Chow, Chris Zhang, Yasi Sun, Yanchao Zhang, Haochen Jiang, Eric Hanchen Liu, Han Huang, Furong Cui, Yuchen Padilla, Oscar Hernan Madrid |
| author_facet | Zhang, Zhi Chow, Chris Zhang, Yasi Sun, Yanchao Zhang, Haochen Jiang, Eric Hanchen Liu, Han Huang, Furong Cui, Yuchen Padilla, Oscar Hernan Madrid |
| contents | Lifelong reinforcement learning (RL) has been developed as a paradigm for extending single-task RL to more realistic, dynamic settings. In lifelong RL, the "life" of an RL agent is modeled as a stream of tasks drawn from a task distribution. We propose EPIC (Empirical PAC-Bayes that Improves Continuously), a novel algorithm designed for lifelong RL using PAC-Bayes theory. EPIC learns a shared policy distribution, referred to as the world policy, which enables rapid adaptation to new tasks while retaining valuable knowledge from previous experiences. Our theoretical analysis establishes a relationship between the algorithm's generalization performance and the number of prior tasks preserved in memory. We also derive the sample complexity of EPIC in terms of RL regret. Extensive experiments on a variety of environments demonstrate that EPIC significantly outperforms existing methods in lifelong RL, offering both theoretical guarantees and practical efficacy through the use of the world policy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_00401 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theory Zhang, Zhi Chow, Chris Zhang, Yasi Sun, Yanchao Zhang, Haochen Jiang, Eric Hanchen Liu, Han Huang, Furong Cui, Yuchen Padilla, Oscar Hernan Madrid Machine Learning Artificial Intelligence 68T05, 68Q32, 68T20 I.2.6; I.2.8; G.3 Lifelong reinforcement learning (RL) has been developed as a paradigm for extending single-task RL to more realistic, dynamic settings. In lifelong RL, the "life" of an RL agent is modeled as a stream of tasks drawn from a task distribution. We propose EPIC (Empirical PAC-Bayes that Improves Continuously), a novel algorithm designed for lifelong RL using PAC-Bayes theory. EPIC learns a shared policy distribution, referred to as the world policy, which enables rapid adaptation to new tasks while retaining valuable knowledge from previous experiences. Our theoretical analysis establishes a relationship between the algorithm's generalization performance and the number of prior tasks preserved in memory. We also derive the sample complexity of EPIC in terms of RL regret. Extensive experiments on a variety of environments demonstrate that EPIC significantly outperforms existing methods in lifelong RL, offering both theoretical guarantees and practical efficacy through the use of the world policy. |
| title | Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theory |
| topic | Machine Learning Artificial Intelligence 68T05, 68Q32, 68T20 I.2.6; I.2.8; G.3 |
| url | https://arxiv.org/abs/2411.00401 |