HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911801176752128 |
|---|---|
| author | Zhang, Mengqi Fu, Yang Ding, Zheng Liu, Sifei Tu, Zhuowen Wang, Xiaolong |
| author_facet | Zhang, Mengqi Fu, Yang Ding, Zheng Liu, Sifei Tu, Zhuowen Wang, Xiaolong |
| contents | 3D hand-object interaction data is scarce due to the hardware constraints in scaling up the data collection process. In this paper, we propose HOIDiffusion for generating realistic and diverse 3D hand-object interaction data. Our model is a conditional diffusion model that takes both the 3D hand-object geometric structure and text description as inputs for image synthesis. This offers a more controllable and realistic synthesis as we can specify the structure and style inputs in a disentangled manner. HOIDiffusion is trained by leveraging a diffusion model pre-trained on large-scale natural images and a few 3D human demonstrations. Beyond controllable image synthesis, we adopt the generated 3D data for learning 6D object pose estimation and show its effectiveness in improving perception systems. Project page: https://mq-zhang1.github.io/HOIDiffusion |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12011 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data Zhang, Mengqi Fu, Yang Ding, Zheng Liu, Sifei Tu, Zhuowen Wang, Xiaolong Computer Vision and Pattern Recognition 3D hand-object interaction data is scarce due to the hardware constraints in scaling up the data collection process. In this paper, we propose HOIDiffusion for generating realistic and diverse 3D hand-object interaction data. Our model is a conditional diffusion model that takes both the 3D hand-object geometric structure and text description as inputs for image synthesis. This offers a more controllable and realistic synthesis as we can specify the structure and style inputs in a disentangled manner. HOIDiffusion is trained by leveraging a diffusion model pre-trained on large-scale natural images and a few 3D human demonstrations. Beyond controllable image synthesis, we adopt the generated 3D data for learning 6D object pose estimation and show its effectiveness in improving perception systems. Project page: https://mq-zhang1.github.io/HOIDiffusion |
| title | HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.12011 |