Hand-Object Interaction Pretraining from Videos
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910601848029184 |
|---|---|
| author | Singh, Himanshu Gaurav Loquercio, Antonio Sferrazza, Carmelo Wu, Jane Qi, Haozhi Abbeel, Pieter Malik, Jitendra |
| author_facet | Singh, Himanshu Gaurav Loquercio, Antonio Sferrazza, Carmelo Wu, Jane Qi, Haozhi Abbeel, Pieter Malik, Jitendra |
| contents | We present an approach to learn general robot manipulation priors from 3D hand-object interaction trajectories. We build a framework to use in-the-wild videos to generate sensorimotor robot trajectories. We do so by lifting both the human hand and the manipulated object in a shared 3D space and retargeting human motions to robot actions. Generative modeling on this data gives us a task-agnostic base policy. This policy captures a general yet flexible manipulation prior. We empirically demonstrate that finetuning this policy, with both reinforcement learning (RL) and behavior cloning (BC), enables sample-efficient adaptation to downstream tasks and simultaneously improves robustness and generalizability compared to prior approaches. Qualitative experiments are available at: \url{https://hgaurav2k.github.io/hop/}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08273 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hand-Object Interaction Pretraining from Videos Singh, Himanshu Gaurav Loquercio, Antonio Sferrazza, Carmelo Wu, Jane Qi, Haozhi Abbeel, Pieter Malik, Jitendra Robotics Artificial Intelligence Computer Vision and Pattern Recognition We present an approach to learn general robot manipulation priors from 3D hand-object interaction trajectories. We build a framework to use in-the-wild videos to generate sensorimotor robot trajectories. We do so by lifting both the human hand and the manipulated object in a shared 3D space and retargeting human motions to robot actions. Generative modeling on this data gives us a task-agnostic base policy. This policy captures a general yet flexible manipulation prior. We empirically demonstrate that finetuning this policy, with both reinforcement learning (RL) and behavior cloning (BC), enables sample-efficient adaptation to downstream tasks and simultaneously improves robustness and generalizability compared to prior approaches. Qualitative experiments are available at: \url{https://hgaurav2k.github.io/hop/}. |
| title | Hand-Object Interaction Pretraining from Videos |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.08273 |