Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916441205243904 |
|---|---|
| author | Lyu, Jiafei Wan, Le Li, Xiu Lu, Zongqing |
| author_facet | Lyu, Jiafei Wan, Le Li, Xiu Lu, Zongqing |
| contents | Recently, there are many efforts attempting to learn useful policies for continuous control in visual reinforcement learning (RL). In this scenario, it is important to learn a generalizable policy, as the testing environment may differ from the training environment, e.g., there exist distractors during deployment. Many practical algorithms are proposed to handle this problem. However, to the best of our knowledge, none of them provide a theoretical understanding of what affects the generalization gap and why their proposed methods work. In this paper, we bridge this issue by theoretically answering the key factors that contribute to the generalization gap when the testing environment has distractors. Our theories indicate that minimizing the representation distance between training and testing environments, which aligns with human intuition, is the most critical for the benefit of reducing the generalization gap. Our theoretical results are supported by the empirical evidence in the DMControl Generalization Benchmark (DMC-GB). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_02701 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence Lyu, Jiafei Wan, Le Li, Xiu Lu, Zongqing Machine Learning Artificial Intelligence Recently, there are many efforts attempting to learn useful policies for continuous control in visual reinforcement learning (RL). In this scenario, it is important to learn a generalizable policy, as the testing environment may differ from the training environment, e.g., there exist distractors during deployment. Many practical algorithms are proposed to handle this problem. However, to the best of our knowledge, none of them provide a theoretical understanding of what affects the generalization gap and why their proposed methods work. In this paper, we bridge this issue by theoretically answering the key factors that contribute to the generalization gap when the testing environment has distractors. Our theories indicate that minimizing the representation distance between training and testing environments, which aligns with human intuition, is the most critical for the benefit of reducing the generalization gap. Our theoretical results are supported by the empirical evidence in the DMControl Generalization Benchmark (DMC-GB). |
| title | Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2402.02701 |