Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.12943 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914926176501760 |
|---|---|
| author | Jain, Vidhi Attarian, Maria Joshi, Nikhil J Wahid, Ayzaan Driess, Danny Vuong, Quan Sanketi, Pannag R Sermanet, Pierre Welker, Stefan Chan, Christine Gilitschenski, Igor Bisk, Yonatan Dwibedi, Debidatta |
| author_facet | Jain, Vidhi Attarian, Maria Joshi, Nikhil J Wahid, Ayzaan Driess, Danny Vuong, Quan Sanketi, Pannag R Sermanet, Pierre Welker, Stefan Chan, Christine Gilitschenski, Igor Bisk, Yonatan Dwibedi, Debidatta |
| contents | Large-scale multi-task robotic manipulation systems often rely on text to specify the task. In this work, we explore whether a robot can learn by observing humans. To do so, the robot must understand a person's intent and perform the inferred task despite differences in the embodiments and environments. We introduce Vid2Robot, an end-to-end video-conditioned policy that takes human videos demonstrating manipulation tasks as input and produces robot actions. Our model is trained with a large dataset of prompt video-robot trajectory pairs to learn unified representations of human and robot actions from videos. Vid2Robot uses cross-attention transformer layers between video features and the current robot state to produce the actions and perform the same task as shown in the video. We use auxiliary contrastive losses to align the prompt and robot video representations for better policies. We evaluate Vid2Robot on real-world robots and observe over 20% improvement over BC-Z when using human prompt videos. Further, we also show cross-object motion transfer ability that enables video-conditioned policies to transfer a motion observed on one object in the prompt video to another object in the robot's own environment. Videos available at https://vid2robot.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12943 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers Jain, Vidhi Attarian, Maria Joshi, Nikhil J Wahid, Ayzaan Driess, Danny Vuong, Quan Sanketi, Pannag R Sermanet, Pierre Welker, Stefan Chan, Christine Gilitschenski, Igor Bisk, Yonatan Dwibedi, Debidatta Robotics Artificial Intelligence Large-scale multi-task robotic manipulation systems often rely on text to specify the task. In this work, we explore whether a robot can learn by observing humans. To do so, the robot must understand a person's intent and perform the inferred task despite differences in the embodiments and environments. We introduce Vid2Robot, an end-to-end video-conditioned policy that takes human videos demonstrating manipulation tasks as input and produces robot actions. Our model is trained with a large dataset of prompt video-robot trajectory pairs to learn unified representations of human and robot actions from videos. Vid2Robot uses cross-attention transformer layers between video features and the current robot state to produce the actions and perform the same task as shown in the video. We use auxiliary contrastive losses to align the prompt and robot video representations for better policies. We evaluate Vid2Robot on real-world robots and observe over 20% improvement over BC-Z when using human prompt videos. Further, we also show cross-object motion transfer ability that enables video-conditioned policies to transfer a motion observed on one object in the prompt video to another object in the robot's own environment. Videos available at https://vid2robot.github.io |
| title | Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2403.12943 |