Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910904697749504 |
|---|---|
| author | Zhou, Jiaming Ma, Teli Lin, Kun-Yu Wang, Zifan Qiu, Ronghe Liang, Junwei |
| author_facet | Zhou, Jiaming Ma, Teli Lin, Kun-Yu Wang, Zifan Qiu, Ronghe Liang, Junwei |
| contents | Learning generalizable visual representations across different embodied environments is essential for effective robotic manipulation in real-world scenarios. However, the limited scale and diversity of robot demonstration data pose a significant challenge. Recent research has explored leveraging large-scale human activity data for pre-training, but the substantial morphological differences between humans and robots introduce a significant human-robot domain discrepancy, hindering the generalization of these models to downstream manipulation tasks. To overcome this, we propose a novel adaptation paradigm that leverages readily available paired human-robot video data to bridge the domain gap. Our method employs a human-robot contrastive alignment loss to align the semantics of human and robot videos, adapting pre-trained models to the robot domain in a parameter-efficient manner. Experiments on 20 simulated tasks across two different benchmarks and five real-world tasks demonstrate significant improvements. These results span both single-task and language-conditioned multi-task settings, evaluated using two different pre-trained models. Compared to existing pre-trained models, our adaptation method improves the average success rate by over 7% across multiple tasks on both simulated benchmarks and real-world evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14235 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulation Zhou, Jiaming Ma, Teli Lin, Kun-Yu Wang, Zifan Qiu, Ronghe Liang, Junwei Computer Vision and Pattern Recognition Robotics Learning generalizable visual representations across different embodied environments is essential for effective robotic manipulation in real-world scenarios. However, the limited scale and diversity of robot demonstration data pose a significant challenge. Recent research has explored leveraging large-scale human activity data for pre-training, but the substantial morphological differences between humans and robots introduce a significant human-robot domain discrepancy, hindering the generalization of these models to downstream manipulation tasks. To overcome this, we propose a novel adaptation paradigm that leverages readily available paired human-robot video data to bridge the domain gap. Our method employs a human-robot contrastive alignment loss to align the semantics of human and robot videos, adapting pre-trained models to the robot domain in a parameter-efficient manner. Experiments on 20 simulated tasks across two different benchmarks and five real-world tasks demonstrate significant improvements. These results span both single-task and language-conditioned multi-task settings, evaluated using two different pre-trained models. Compared to existing pre-trained models, our adaptation method improves the average success rate by over 7% across multiple tasks on both simulated benchmarks and real-world evaluations. |
| title | Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulation |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2406.14235 |