St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912333116211200 |
|---|---|
| author | Feng, Haiwen Zhang, Junyi Wang, Qianqian Ye, Yufei Yu, Pengcheng Black, Michael J. Darrell, Trevor Kanazawa, Angjoo |
| author_facet | Feng, Haiwen Zhang, Junyi Wang, Qianqian Ye, Yufei Yu, Pengcheng Black, Michael J. Darrell, Trevor Kanazawa, Angjoo |
| contents | Dynamic 3D reconstruction and point tracking in videos are typically treated as separate tasks, despite their deep connection. We propose St4RTrack, a feed-forward framework that simultaneously reconstructs and tracks dynamic video content in a world coordinate frame from RGB inputs. This is achieved by predicting two appropriately defined pointmaps for a pair of frames captured at different moments. Specifically, we predict both pointmaps at the same moment, in the same world, capturing both static and dynamic scene geometry while maintaining 3D correspondences. Chaining these predictions through the video sequence with respect to a reference frame naturally computes long-range correspondences, effectively combining 3D reconstruction with 3D tracking. Unlike prior methods that rely heavily on 4D ground truth supervision, we employ a novel adaptation scheme based on a reprojection loss. We establish a new extensive benchmark for world-frame reconstruction and tracking, demonstrating the effectiveness and efficiency of our unified, data-driven framework. Our code, model, and benchmark will be released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_13152 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World Feng, Haiwen Zhang, Junyi Wang, Qianqian Ye, Yufei Yu, Pengcheng Black, Michael J. Darrell, Trevor Kanazawa, Angjoo Computer Vision and Pattern Recognition Dynamic 3D reconstruction and point tracking in videos are typically treated as separate tasks, despite their deep connection. We propose St4RTrack, a feed-forward framework that simultaneously reconstructs and tracks dynamic video content in a world coordinate frame from RGB inputs. This is achieved by predicting two appropriately defined pointmaps for a pair of frames captured at different moments. Specifically, we predict both pointmaps at the same moment, in the same world, capturing both static and dynamic scene geometry while maintaining 3D correspondences. Chaining these predictions through the video sequence with respect to a reference frame naturally computes long-range correspondences, effectively combining 3D reconstruction with 3D tracking. Unlike prior methods that rely heavily on 4D ground truth supervision, we employ a novel adaptation scheme based on a reprojection loss. We establish a new extensive benchmark for world-frame reconstruction and tracking, demonstrating the effectiveness and efficiency of our unified, data-driven framework. Our code, model, and benchmark will be released. |
| title | St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.13152 |