Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.06182 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910738947244032 |
|---|---|
| author | You, Zeng Wen, Zhiquan Chen, Yaofo Li, Xin Zeng, Runhao Wang, Yaowei Tan, Mingkui |
| author_facet | You, Zeng Wen, Zhiquan Chen, Yaofo Li, Xin Zeng, Runhao Wang, Yaowei Tan, Mingkui |
| contents | Long video understanding has become a critical task in computer vision, driving advancements across numerous applications from surveillance to content retrieval. Existing video understanding methods suffer from two challenges when dealing with long video understanding: intricate long-context relationship modeling and interference from redundancy. To tackle these challenges, we introduce Fine-Detailed Video Story generation (FDVS), which interprets long videos into detailed textual representations. Specifically, to achieve fine-grained modeling of long-temporal content, we propose a Bottom-up Video Interpretation Mechanism that progressively interprets video content from clips to video. To avoid interference from redundant information in videos, we introduce a Semantic Redundancy Reduction mechanism that removes redundancy at both the visual and textual levels. Our method transforms long videos into hierarchical textual representations that contain multi-granularity information of the video. With these representations, FDVS is applicable to various tasks without any fine-tuning. We evaluate the proposed method across eight datasets spanning three tasks. The performance demonstrates the effectiveness and versatility of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_06182 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Long Video Understanding via Fine-detailed Video Story Generation You, Zeng Wen, Zhiquan Chen, Yaofo Li, Xin Zeng, Runhao Wang, Yaowei Tan, Mingkui Computer Vision and Pattern Recognition Long video understanding has become a critical task in computer vision, driving advancements across numerous applications from surveillance to content retrieval. Existing video understanding methods suffer from two challenges when dealing with long video understanding: intricate long-context relationship modeling and interference from redundancy. To tackle these challenges, we introduce Fine-Detailed Video Story generation (FDVS), which interprets long videos into detailed textual representations. Specifically, to achieve fine-grained modeling of long-temporal content, we propose a Bottom-up Video Interpretation Mechanism that progressively interprets video content from clips to video. To avoid interference from redundant information in videos, we introduce a Semantic Redundancy Reduction mechanism that removes redundancy at both the visual and textual levels. Our method transforms long videos into hierarchical textual representations that contain multi-granularity information of the video. With these representations, FDVS is applicable to various tasks without any fine-tuning. We evaluate the proposed method across eight datasets spanning three tasks. The performance demonstrates the effectiveness and versatility of our method. |
| title | Towards Long Video Understanding via Fine-detailed Video Story Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.06182 |