Pre-trained Visual Representations Generalize Where it Matters in Model-Based Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912588832440320 |
|---|---|
| author | Jones, Scott Zhou, Liyou Pattinson, Sebastian W. |
| author_facet | Jones, Scott Zhou, Liyou Pattinson, Sebastian W. |
| contents | In visuomotor policy learning, the control policy for the robotic agent is derived directly from visual inputs. The typical approach, where a policy and vision encoder are trained jointly from scratch, generalizes poorly to novel visual scene changes. Using pre-trained vision models (PVMs) to inform a policy network improves robustness in model-free reinforcement learning (MFRL). Recent developments in Model-based reinforcement learning (MBRL) suggest that MBRL is more sample-efficient than MFRL. However, counterintuitively, existing work has found PVMs to be ineffective in MBRL. Here, we investigate PVM's effectiveness in MBRL, specifically on generalization under visual domain shifts. We show that, in scenarios with severe shifts, PVMs perform much better than a baseline model trained from scratch. We further investigate the effects of varying levels of fine-tuning of PVMs. Our results show that partial fine-tuning can maintain the highest average task performance under the most extreme distribution shifts. Our results demonstrate that PVMs are highly successful in promoting robustness in visual policy learning, providing compelling evidence for their wider adoption in model-based robotic learning applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12531 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pre-trained Visual Representations Generalize Where it Matters in Model-Based Reinforcement Learning Jones, Scott Zhou, Liyou Pattinson, Sebastian W. Robotics Artificial Intelligence Machine Learning Systems and Control 68T07, 68T40 (Primary) 93C85, 62L20 (Secondary) I.2.6; I.2.9; I.4.8; F.2.2 In visuomotor policy learning, the control policy for the robotic agent is derived directly from visual inputs. The typical approach, where a policy and vision encoder are trained jointly from scratch, generalizes poorly to novel visual scene changes. Using pre-trained vision models (PVMs) to inform a policy network improves robustness in model-free reinforcement learning (MFRL). Recent developments in Model-based reinforcement learning (MBRL) suggest that MBRL is more sample-efficient than MFRL. However, counterintuitively, existing work has found PVMs to be ineffective in MBRL. Here, we investigate PVM's effectiveness in MBRL, specifically on generalization under visual domain shifts. We show that, in scenarios with severe shifts, PVMs perform much better than a baseline model trained from scratch. We further investigate the effects of varying levels of fine-tuning of PVMs. Our results show that partial fine-tuning can maintain the highest average task performance under the most extreme distribution shifts. Our results demonstrate that PVMs are highly successful in promoting robustness in visual policy learning, providing compelling evidence for their wider adoption in model-based robotic learning applications. |
| title | Pre-trained Visual Representations Generalize Where it Matters in Model-Based Reinforcement Learning |
| topic | Robotics Artificial Intelligence Machine Learning Systems and Control 68T07, 68T40 (Primary) 93C85, 62L20 (Secondary) I.2.6; I.2.9; I.4.8; F.2.2 |
| url | https://arxiv.org/abs/2509.12531 |