Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911332789387264 |
|---|---|
| author | Wang, Yinhuai Yu, Runyi Tsui, Hok Wai Lin, Xiaoyi Zhang, Hui Zhao, Qihan Fan, Ke Li, Miao Song, Jie Wang, Jingbo Chen, Qifeng Tan, Ping |
| author_facet | Wang, Yinhuai Yu, Runyi Tsui, Hok Wai Lin, Xiaoyi Zhang, Hui Zhao, Qihan Fan, Ke Li, Miao Song, Jie Wang, Jingbo Chen, Qifeng Tan, Ping |
| contents | We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOP, a Hand-Object Planner, which can synthesize diverse hand-object trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning, delivering a generalizable controller conditioned on target hand-object states. Our method extends to diverse object shapes and hand morphologies. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_19583 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations Wang, Yinhuai Yu, Runyi Tsui, Hok Wai Lin, Xiaoyi Zhang, Hui Zhao, Qihan Fan, Ke Li, Miao Song, Jie Wang, Jingbo Chen, Qifeng Tan, Ping Robotics Graphics We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOP, a Hand-Object Planner, which can synthesize diverse hand-object trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning, delivering a generalizable controller conditioned on target hand-object states. Our method extends to diverse object shapes and hand morphologies. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation. |
| title | Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations |
| topic | Robotics Graphics |
| url | https://arxiv.org/abs/2512.19583 |