Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909324426608640 |
|---|---|
| author | Bharadhwaj, Homanga Dwibedi, Debidatta Gupta, Abhinav Tulsiani, Shubham Doersch, Carl Xiao, Ted Shah, Dhruv Xia, Fei Sadigh, Dorsa Kirmani, Sean |
| author_facet | Bharadhwaj, Homanga Dwibedi, Debidatta Gupta, Abhinav Tulsiani, Shubham Doersch, Carl Xiao, Ted Shah, Dhruv Xia, Fei Sadigh, Dorsa Kirmani, Sean |
| contents | How can robot manipulation policies generalize to novel tasks involving unseen object types and new motions? In this paper, we provide a solution in terms of predicting motion information from web data through human video generation and conditioning a robot policy on the generated video. Instead of attempting to scale robot data collection which is expensive, we show how we can leverage video generation models trained on easily available web data, for enabling generalization. Our approach Gen2Act casts language-conditioned manipulation as zero-shot human video generation followed by execution with a single policy conditioned on the generated video. To train the policy, we use an order of magnitude less robot interaction data compared to what the video prediction model was trained on. Gen2Act doesn't require fine-tuning the video model at all and we directly use a pre-trained model for generating human videos. Our results on diverse real-world scenarios show how Gen2Act enables manipulating unseen object types and performing novel motions for tasks not present in the robot data. Videos are at https://homangab.github.io/gen2act/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_16283 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation Bharadhwaj, Homanga Dwibedi, Debidatta Gupta, Abhinav Tulsiani, Shubham Doersch, Carl Xiao, Ted Shah, Dhruv Xia, Fei Sadigh, Dorsa Kirmani, Sean Robotics Computer Vision and Pattern Recognition Machine Learning Image and Video Processing How can robot manipulation policies generalize to novel tasks involving unseen object types and new motions? In this paper, we provide a solution in terms of predicting motion information from web data through human video generation and conditioning a robot policy on the generated video. Instead of attempting to scale robot data collection which is expensive, we show how we can leverage video generation models trained on easily available web data, for enabling generalization. Our approach Gen2Act casts language-conditioned manipulation as zero-shot human video generation followed by execution with a single policy conditioned on the generated video. To train the policy, we use an order of magnitude less robot interaction data compared to what the video prediction model was trained on. Gen2Act doesn't require fine-tuning the video model at all and we directly use a pre-trained model for generating human videos. Our results on diverse real-world scenarios show how Gen2Act enables manipulating unseen object types and performing novel motions for tasks not present in the robot data. Videos are at https://homangab.github.io/gen2act/ |
| title | Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation |
| topic | Robotics Computer Vision and Pattern Recognition Machine Learning Image and Video Processing |
| url | https://arxiv.org/abs/2409.16283 |