ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913622510272512 |
|---|---|
| author | Pang, Youxin Shao, Ruizhi Zhang, Jiajun Tu, Hanzhang Liu, Yun Zhou, Boyao Zhang, Hongwen Liu, Yebin |
| author_facet | Pang, Youxin Shao, Ruizhi Zhang, Jiajun Tu, Hanzhang Liu, Yun Zhou, Boyao Zhang, Hongwen Liu, Yebin |
| contents | In this paper, we introduce ManiVideo, a novel method for generating consistent and temporally coherent bimanual hand-object manipulation videos from given motion sequences of hands and objects. The core idea of ManiVideo is the construction of a multi-layer occlusion (MLO) representation that learns 3D occlusion relationships from occlusion-free normal maps and occlusion confidence maps. By embedding the MLO structure into the UNet in two forms, the model enhances the 3D consistency of dexterous hand-object manipulation. To further achieve the generalizable grasping of objects, we integrate Objaverse, a large-scale 3D object dataset, to address the scarcity of video data, thereby facilitating the learning of extensive object consistency. Additionally, we propose an innovative training strategy that effectively integrates multiple datasets, supporting downstream tasks such as human-centric hand-object manipulation video generation. Through extensive experiments, we demonstrate that our approach not only achieves video generation with plausible hand-object interaction and generalizable objects, but also outperforms existing SOTA methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_16212 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping Pang, Youxin Shao, Ruizhi Zhang, Jiajun Tu, Hanzhang Liu, Yun Zhou, Boyao Zhang, Hongwen Liu, Yebin Computer Vision and Pattern Recognition In this paper, we introduce ManiVideo, a novel method for generating consistent and temporally coherent bimanual hand-object manipulation videos from given motion sequences of hands and objects. The core idea of ManiVideo is the construction of a multi-layer occlusion (MLO) representation that learns 3D occlusion relationships from occlusion-free normal maps and occlusion confidence maps. By embedding the MLO structure into the UNet in two forms, the model enhances the 3D consistency of dexterous hand-object manipulation. To further achieve the generalizable grasping of objects, we integrate Objaverse, a large-scale 3D object dataset, to address the scarcity of video data, thereby facilitating the learning of extensive object consistency. Additionally, we propose an innovative training strategy that effectively integrates multiple datasets, supporting downstream tasks such as human-centric hand-object manipulation video generation. Through extensive experiments, we demonstrate that our approach not only achieves video generation with plausible hand-object interaction and generalizable objects, but also outperforms existing SOTA methods. |
| title | ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.16212 |