Feature Extractor or Decision Maker: Rethinking the Role of Visual Encoders in Visuomotor Policies
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909609802858496 |
|---|---|
| author | Wang, Ruiyu Zhuang, Zheyu Jin, Shutong Ingelhag, Nils Kragic, Danica Pokorny, Florian T. |
| author_facet | Wang, Ruiyu Zhuang, Zheyu Jin, Shutong Ingelhag, Nils Kragic, Danica Pokorny, Florian T. |
| contents | An end-to-end (E2E) visuomotor policy is typically treated as a unified whole, but recent approaches using out-of-domain (OOD) data to pretrain the visual encoder have cleanly separated the visual encoder from the network, with the remainder referred to as the policy. We propose Visual Alignment Testing, an experimental framework designed to evaluate the validity of this functional separation. Our results indicate that in E2E-trained models, visual encoders actively contribute to decision-making resulting from motor data supervision, contradicting the assumed functional separation. In contrast, OOD-pretrained models, where encoders lack this capability, experience an average performance drop of 42\% in our benchmark results, compared to the state-of-the-art performance achieved by E2E policies. We believe this initial exploration of visual encoders' role can provide a first step towards guiding future pretraining methods to address their decision-making ability, such as developing task-conditioned or context-aware encoders. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_20248 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Feature Extractor or Decision Maker: Rethinking the Role of Visual Encoders in Visuomotor Policies Wang, Ruiyu Zhuang, Zheyu Jin, Shutong Ingelhag, Nils Kragic, Danica Pokorny, Florian T. Robotics An end-to-end (E2E) visuomotor policy is typically treated as a unified whole, but recent approaches using out-of-domain (OOD) data to pretrain the visual encoder have cleanly separated the visual encoder from the network, with the remainder referred to as the policy. We propose Visual Alignment Testing, an experimental framework designed to evaluate the validity of this functional separation. Our results indicate that in E2E-trained models, visual encoders actively contribute to decision-making resulting from motor data supervision, contradicting the assumed functional separation. In contrast, OOD-pretrained models, where encoders lack this capability, experience an average performance drop of 42\% in our benchmark results, compared to the state-of-the-art performance achieved by E2E policies. We believe this initial exploration of visual encoders' role can provide a first step towards guiding future pretraining methods to address their decision-making ability, such as developing task-conditioned or context-aware encoders. |
| title | Feature Extractor or Decision Maker: Rethinking the Role of Visual Encoders in Visuomotor Policies |
| topic | Robotics |
| url | https://arxiv.org/abs/2409.20248 |