When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917872167550976 |
|---|---|
| author | Mu, Tongzhou Li, Zhaoyang Strzelecki, Stanisław Wiktor Yuan, Xiu Yao, Yunchao Liang, Litian Su, Hao |
| author_facet | Mu, Tongzhou Li, Zhaoyang Strzelecki, Stanisław Wiktor Yuan, Xiu Yao, Yunchao Liang, Litian Su, Hao |
| contents | Learning policies from high-dimensional visual inputs, such as pixels and point clouds, is crucial in various applications. Visual reinforcement learning is a promising approach that directly trains policies from visual observations, although it faces challenges in sample efficiency and computational costs. This study conducts an empirical comparison of State-to-Visual DAgger, a two-stage framework that initially trains a state policy before adopting online imitation to learn a visual policy, and Visual RL across a diverse set of tasks. We evaluate both methods across 16 tasks from three benchmarks, focusing on their asymptotic performance, sample efficiency, and computational costs. Surprisingly, our findings reveal that State-to-Visual DAgger does not universally outperform Visual RL but shows significant advantages in challenging tasks, offering more consistent performance. In contrast, its benefits in sample efficiency are less pronounced, although it often reduces the overall wall-clock time required for training. Based on our findings, we provide recommendations for practitioners and hope that our results contribute valuable perspectives for future research in visual policy learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13662 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning? Mu, Tongzhou Li, Zhaoyang Strzelecki, Stanisław Wiktor Yuan, Xiu Yao, Yunchao Liang, Litian Su, Hao Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robotics Learning policies from high-dimensional visual inputs, such as pixels and point clouds, is crucial in various applications. Visual reinforcement learning is a promising approach that directly trains policies from visual observations, although it faces challenges in sample efficiency and computational costs. This study conducts an empirical comparison of State-to-Visual DAgger, a two-stage framework that initially trains a state policy before adopting online imitation to learn a visual policy, and Visual RL across a diverse set of tasks. We evaluate both methods across 16 tasks from three benchmarks, focusing on their asymptotic performance, sample efficiency, and computational costs. Surprisingly, our findings reveal that State-to-Visual DAgger does not universally outperform Visual RL but shows significant advantages in challenging tasks, offering more consistent performance. In contrast, its benefits in sample efficiency are less pronounced, although it often reduces the overall wall-clock time required for training. Based on our findings, we provide recommendations for practitioners and hope that our results contribute valuable perspectives for future research in visual policy learning. |
| title | When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning? |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robotics |
| url | https://arxiv.org/abs/2412.13662 |