Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.22193 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915907988619264 |
|---|---|
| author | Gao, Mingju Yang, Kaisen Gao, Huan-ang Li, Bohan Ding, Ao Li, Wenyi Yu, Yangcheng Liu, Jinkun Xu, Shaocong Niu, Yike Chi, Haohan Chen, Hao Tang, Hao Zhang, Yu Yi, Li Zhao, Hao |
| author_facet | Gao, Mingju Yang, Kaisen Gao, Huan-ang Li, Bohan Ding, Ao Li, Wenyi Yu, Yangcheng Liu, Jinkun Xu, Shaocong Niu, Yike Chi, Haohan Chen, Hao Tang, Hao Zhang, Yu Yi, Li Zhao, Hao |
| contents | Hand-object interaction (HOI) reconstruction and synthesis are becoming central to embodied AI and AR/VR. Yet, despite rapid progress, existing HOI generation research remains fragmented across three disjoint tracks: (1) pose-only synthesis that predicts MANO trajectories without producing pixels; (2) single-image HOI generation that hallucinates appearance from masks or 2D cues but lacks dynamics; and (3) video generation methods that require both the entire pose sequence and the ground-truth first frame as inputs, preventing true sim-to-real deployment. Inspired by the philosophy of Joo et al. (2018), we think that HOI generation requires a unified engine that brings together pose, appearance, and motion within one coherent framework. Thus we introduce PAM: a Pose-Appearance-Motion Engine for controllable HOI video generation. The performance of our engine is validated by: (1) On DexYCB, we obtain an FVD of 29.13 (vs. 38.83 for InterDyn), and MPJPE of 19.37 mm (vs. 30.05 mm for CosHand), while generating higher-resolution 480x720 videos compared to 256x256 and 256x384 baselines. (2) On OAKINK2, our full multi-condition model improves FVD from 68.76 to 46.31. (3) An ablation over input conditions on DexYCB shows that combining depth, segmentation, and keypoints consistently yields the best results. (4) For a downstream hand pose estimation task using SimpleHand, augmenting training with 3,400 synthetic videos (207k frames) allows a model trained on only 50% of the real data plus our synthetic data to match the 100% real baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_22193 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generation Gao, Mingju Yang, Kaisen Gao, Huan-ang Li, Bohan Ding, Ao Li, Wenyi Yu, Yangcheng Liu, Jinkun Xu, Shaocong Niu, Yike Chi, Haohan Chen, Hao Tang, Hao Zhang, Yu Yi, Li Zhao, Hao Computer Vision and Pattern Recognition Hand-object interaction (HOI) reconstruction and synthesis are becoming central to embodied AI and AR/VR. Yet, despite rapid progress, existing HOI generation research remains fragmented across three disjoint tracks: (1) pose-only synthesis that predicts MANO trajectories without producing pixels; (2) single-image HOI generation that hallucinates appearance from masks or 2D cues but lacks dynamics; and (3) video generation methods that require both the entire pose sequence and the ground-truth first frame as inputs, preventing true sim-to-real deployment. Inspired by the philosophy of Joo et al. (2018), we think that HOI generation requires a unified engine that brings together pose, appearance, and motion within one coherent framework. Thus we introduce PAM: a Pose-Appearance-Motion Engine for controllable HOI video generation. The performance of our engine is validated by: (1) On DexYCB, we obtain an FVD of 29.13 (vs. 38.83 for InterDyn), and MPJPE of 19.37 mm (vs. 30.05 mm for CosHand), while generating higher-resolution 480x720 videos compared to 256x256 and 256x384 baselines. (2) On OAKINK2, our full multi-condition model improves FVD from 68.76 to 46.31. (3) An ablation over input conditions on DexYCB shows that combining depth, segmentation, and keypoints consistently yields the best results. (4) For a downstream hand pose estimation task using SimpleHand, augmenting training with 3,400 synthetic videos (207k frames) allows a model trained on only 50% of the real data plus our synthetic data to match the 100% real baseline. |
| title | PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.22193 |