Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.10332 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910887773732864 |
|---|---|
| author | Wu, Yongliang Hu, Xinting Sun, Yuyang Zhou, Yizhou Zhu, Wenbo Rao, Fengyun Schiele, Bernt Yang, Xu |
| author_facet | Wu, Yongliang Hu, Xinting Sun, Yuyang Zhou, Yizhou Zhu, Wenbo Rao, Fengyun Schiele, Bernt Yang, Xu |
| contents | Video Large Language Models (Vid-LLMs) have made remarkable advancements in comprehending video content for QA dialogue. However, they struggle to extend this visual understanding to tasks requiring precise temporal localization, known as Video Temporal Grounding (VTG). To address this gap, we introduce Number-Prompt (NumPro), a novel method that empowers Vid-LLMs to bridge visual comprehension with temporal grounding by adding unique numerical identifiers to each video frame. Treating a video as a sequence of numbered frame images, NumPro transforms VTG into an intuitive process: flipping through manga panels in sequence. This allows Vid-LLMs to "read" event timelines, accurately linking visual content with corresponding temporal information. Our experiments demonstrate that NumPro significantly boosts VTG performance of top-tier Vid-LLMs without additional computational cost. Furthermore, fine-tuning on a NumPro-enhanced dataset defines a new state-of-the-art for VTG, surpassing previous top-performing methods by up to 6.9\% in mIoU for moment retrieval and 8.5\% in mAP for highlight detection. The code will be available at https://github.com/yongliang-wu/NumPro. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_10332 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Number it: Temporal Grounding Videos like Flipping Manga Wu, Yongliang Hu, Xinting Sun, Yuyang Zhou, Yizhou Zhu, Wenbo Rao, Fengyun Schiele, Bernt Yang, Xu Computer Vision and Pattern Recognition Video Large Language Models (Vid-LLMs) have made remarkable advancements in comprehending video content for QA dialogue. However, they struggle to extend this visual understanding to tasks requiring precise temporal localization, known as Video Temporal Grounding (VTG). To address this gap, we introduce Number-Prompt (NumPro), a novel method that empowers Vid-LLMs to bridge visual comprehension with temporal grounding by adding unique numerical identifiers to each video frame. Treating a video as a sequence of numbered frame images, NumPro transforms VTG into an intuitive process: flipping through manga panels in sequence. This allows Vid-LLMs to "read" event timelines, accurately linking visual content with corresponding temporal information. Our experiments demonstrate that NumPro significantly boosts VTG performance of top-tier Vid-LLMs without additional computational cost. Furthermore, fine-tuning on a NumPro-enhanced dataset defines a new state-of-the-art for VTG, surpassing previous top-performing methods by up to 6.9\% in mIoU for moment retrieval and 8.5\% in mAP for highlight detection. The code will be available at https://github.com/yongliang-wu/NumPro. |
| title | Number it: Temporal Grounding Videos like Flipping Manga |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.10332 |