Scene Summarization: Clustering Scene Videos into Spatially Diverse Frames
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914167530717184 |
|---|---|
| author | Chen, Chao Zhu, Mingzhi Singh, Ankush Pratap Yan, Yu Juefei-Xu, Felix Feng, Chen |
| author_facet | Chen, Chao Zhu, Mingzhi Singh, Ankush Pratap Yan, Yu Juefei-Xu, Felix Feng, Chen |
| contents | Humans are remarkably efficient at forming spatial understanding from just a few visual observations. When browsing real estate or navigating unfamiliar spaces, they intuitively select a small set of views that summarize the spatial layout. Inspired by this ability, we introduce scene summarization, the task of condensing long, continuous scene videos into a compact set of spatially diverse keyframes that facilitate global spatial reasoning. Unlike conventional video summarization-which focuses on user-edited, fragmented clips and often ignores spatial continuity-our goal is to mimic how humans abstract spatial layout from sparse views. We propose SceneSum, a two-stage self-supervised pipeline that first clusters video frames using visual place recognition to promote spatial diversity, then selects representative keyframes from each cluster under resource constraints. When camera trajectories are available, a lightweight supervised loss further refines clustering and selection. Experiments on real and simulated indoor datasets show that SceneSum produces more spatially informative summaries and outperforms existing video summarization baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_17940 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Scene Summarization: Clustering Scene Videos into Spatially Diverse Frames Chen, Chao Zhu, Mingzhi Singh, Ankush Pratap Yan, Yu Juefei-Xu, Felix Feng, Chen Computer Vision and Pattern Recognition Humans are remarkably efficient at forming spatial understanding from just a few visual observations. When browsing real estate or navigating unfamiliar spaces, they intuitively select a small set of views that summarize the spatial layout. Inspired by this ability, we introduce scene summarization, the task of condensing long, continuous scene videos into a compact set of spatially diverse keyframes that facilitate global spatial reasoning. Unlike conventional video summarization-which focuses on user-edited, fragmented clips and often ignores spatial continuity-our goal is to mimic how humans abstract spatial layout from sparse views. We propose SceneSum, a two-stage self-supervised pipeline that first clusters video frames using visual place recognition to promote spatial diversity, then selects representative keyframes from each cluster under resource constraints. When camera trajectories are available, a lightweight supervised loss further refines clustering and selection. Experiments on real and simulated indoor datasets show that SceneSum produces more spatially informative summaries and outperforms existing video summarization baselines. |
| title | Scene Summarization: Clustering Scene Videos into Spatially Diverse Frames |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.17940 |