CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913547705909248 |
|---|---|
| author | Karaev, Nikita Makarov, Iurii Wang, Jianyuan Neverova, Natalia Vedaldi, Andrea Rupprecht, Christian |
| author_facet | Karaev, Nikita Makarov, Iurii Wang, Jianyuan Neverova, Natalia Vedaldi, Andrea Rupprecht, Christian |
| contents | Most state-of-the-art point trackers are trained on synthetic data due to the difficulty of annotating real videos for this task. However, this can result in suboptimal performance due to the statistical gap between synthetic and real videos. In order to understand these issues better, we introduce CoTracker3, comprising a new tracking model and a new semi-supervised training recipe. This allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers. The new model eliminates or simplifies components from previous trackers, resulting in a simpler and often smaller architecture. This training scheme is much simpler than prior work and achieves better results using 1,000 times less data. We further study the scaling behaviour to understand the impact of using more real unsupervised data in point tracking. The model is available in online and offline variants and reliably tracks visible and occluded points. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11831 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos Karaev, Nikita Makarov, Iurii Wang, Jianyuan Neverova, Natalia Vedaldi, Andrea Rupprecht, Christian Computer Vision and Pattern Recognition Most state-of-the-art point trackers are trained on synthetic data due to the difficulty of annotating real videos for this task. However, this can result in suboptimal performance due to the statistical gap between synthetic and real videos. In order to understand these issues better, we introduce CoTracker3, comprising a new tracking model and a new semi-supervised training recipe. This allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers. The new model eliminates or simplifies components from previous trackers, resulting in a simpler and often smaller architecture. This training scheme is much simpler than prior work and achieves better results using 1,000 times less data. We further study the scaling behaviour to understand the impact of using more real unsupervised data in point tracking. The model is available in online and offline variants and reliably tracks visible and occluded points. |
| title | CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.11831 |