Towards Understanding Camera Motions in Any Video
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909759207112704 |
|---|---|
| author | Lin, Zhiqiu Cen, Siyuan Jiang, Daniel Karhade, Jay Wang, Hewei Mitra, Chancharik Ling, Tiffany Huang, Yuhan Liu, Sifan Chen, Mingyu Zawar, Rushikesh Bai, Xue Du, Yilun Gan, Chuang Ramanan, Deva |
| author_facet | Lin, Zhiqiu Cen, Siyuan Jiang, Daniel Karhade, Jay Wang, Hewei Mitra, Chancharik Ling, Tiffany Huang, Yuhan Liu, Sifan Chen, Mingyu Zawar, Rushikesh Bai, Xue Du, Yilun Gan, Chuang Ramanan, Deva |
| contents | We introduce CameraBench, a large-scale dataset and benchmark designed to assess and improve camera motion understanding. CameraBench consists of ~3,000 diverse internet videos, annotated by experts through a rigorous multi-stage quality control process. One of our contributions is a taxonomy of camera motion primitives, designed in collaboration with cinematographers. We find, for example, that some motions like "follow" (or tracking) require understanding scene content like moving subjects. We conduct a large-scale human study to quantify human annotation performance, revealing that domain expertise and tutorial-based training can significantly enhance accuracy. For example, a novice may confuse zoom-in (a change of intrinsics) with translating forward (a change of extrinsics), but can be trained to differentiate the two. Using CameraBench, we evaluate Structure-from-Motion (SfM) and Video-Language Models (VLMs), finding that SfM models struggle to capture semantic primitives that depend on scene content, while VLMs struggle to capture geometric primitives that require precise estimation of trajectories. We then fine-tune a generative VLM on CameraBench to achieve the best of both worlds and showcase its applications, including motion-augmented captioning, video question answering, and video-text retrieval. We hope our taxonomy, benchmark, and tutorials will drive future efforts towards the ultimate goal of understanding camera motions in any video. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15376 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Understanding Camera Motions in Any Video Lin, Zhiqiu Cen, Siyuan Jiang, Daniel Karhade, Jay Wang, Hewei Mitra, Chancharik Ling, Tiffany Huang, Yuhan Liu, Sifan Chen, Mingyu Zawar, Rushikesh Bai, Xue Du, Yilun Gan, Chuang Ramanan, Deva Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Multimedia We introduce CameraBench, a large-scale dataset and benchmark designed to assess and improve camera motion understanding. CameraBench consists of ~3,000 diverse internet videos, annotated by experts through a rigorous multi-stage quality control process. One of our contributions is a taxonomy of camera motion primitives, designed in collaboration with cinematographers. We find, for example, that some motions like "follow" (or tracking) require understanding scene content like moving subjects. We conduct a large-scale human study to quantify human annotation performance, revealing that domain expertise and tutorial-based training can significantly enhance accuracy. For example, a novice may confuse zoom-in (a change of intrinsics) with translating forward (a change of extrinsics), but can be trained to differentiate the two. Using CameraBench, we evaluate Structure-from-Motion (SfM) and Video-Language Models (VLMs), finding that SfM models struggle to capture semantic primitives that depend on scene content, while VLMs struggle to capture geometric primitives that require precise estimation of trajectories. We then fine-tune a generative VLM on CameraBench to achieve the best of both worlds and showcase its applications, including motion-augmented captioning, video question answering, and video-text retrieval. We hope our taxonomy, benchmark, and tutorials will drive future efforts towards the ultimate goal of understanding camera motions in any video. |
| title | Towards Understanding Camera Motions in Any Video |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Multimedia |
| url | https://arxiv.org/abs/2504.15376 |