FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912144852779008 |
|---|---|
| author | Chen, Kefan Min, Chaerin Zhang, Linguang Hampali, Shreyas Keskin, Cem Sridhar, Srinath |
| author_facet | Chen, Kefan Min, Chaerin Zhang, Linguang Hampali, Shreyas Keskin, Cem Sridhar, Srinath |
| contents | Despite remarkable progress in image generation models, generating realistic hands remains a persistent challenge due to their complex articulation, varying viewpoints, and frequent occlusions. We present FoundHand, a large-scale domain-specific diffusion model for synthesizing single and dual hand images. To train our model, we introduce FoundHand-10M, a large-scale hand dataset with 2D keypoints and segmentation mask annotations. Our insight is to use 2D hand keypoints as a universal representation that encodes both hand articulation and camera viewpoint. FoundHand learns from image pairs to capture physically plausible hand articulations, natively enables precise control through 2D keypoints, and supports appearance control. Our model exhibits core capabilities that include the ability to repose hands, transfer hand appearance, and even synthesize novel views. This leads to zero-shot capabilities for fixing malformed hands in previously generated images, or synthesizing hand video sequences. We present extensive experiments and evaluations that demonstrate state-of-the-art performance of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02690 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation Chen, Kefan Min, Chaerin Zhang, Linguang Hampali, Shreyas Keskin, Cem Sridhar, Srinath Computer Vision and Pattern Recognition Despite remarkable progress in image generation models, generating realistic hands remains a persistent challenge due to their complex articulation, varying viewpoints, and frequent occlusions. We present FoundHand, a large-scale domain-specific diffusion model for synthesizing single and dual hand images. To train our model, we introduce FoundHand-10M, a large-scale hand dataset with 2D keypoints and segmentation mask annotations. Our insight is to use 2D hand keypoints as a universal representation that encodes both hand articulation and camera viewpoint. FoundHand learns from image pairs to capture physically plausible hand articulations, natively enables precise control through 2D keypoints, and supports appearance control. Our model exhibits core capabilities that include the ability to repose hands, transfer hand appearance, and even synthesize novel views. This leads to zero-shot capabilities for fixing malformed hands in previously generated images, or synthesizing hand video sequences. We present extensive experiments and evaluations that demonstrate state-of-the-art performance of our method. |
| title | FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.02690 |