Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917847975854080 |
|---|---|
| author | Deng, Andong Gao, Zhongpai Choudhuri, Anwesa Planche, Benjamin Zheng, Meng Wang, Bin Chen, Terrence Chen, Chen Wu, Ziyan |
| author_facet | Deng, Andong Gao, Zhongpai Choudhuri, Anwesa Planche, Benjamin Zheng, Meng Wang, Bin Chen, Terrence Chen, Chen Wu, Ziyan |
| contents | Temporal awareness is essential for video large language models (LLMs) to understand and reason about events within long videos, enabling applications like dense video captioning and temporal video grounding in a unified system. However, the scarcity of long videos with detailed captions and precise temporal annotations limits their temporal awareness. In this paper, we propose Seq2Time, a data-oriented training paradigm that leverages sequences of images and short video clips to enhance temporal awareness in long videos. By converting sequence positions into temporal annotations, we transform large-scale image and clip captioning datasets into sequences that mimic the temporal structure of long videos, enabling self-supervised training with abundant time-sensitive data. To enable sequence-to-time knowledge transfer, we introduce a novel time representation that unifies positional information across image sequences, clip sequences, and long videos. Experiments demonstrate the effectiveness of our method, achieving a 27.6% improvement in F1 score and 44.8% in CIDEr on the YouCook2 benchmark and a 14.7% increase in recall on the Charades-STA benchmark compared to the baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16932 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding Deng, Andong Gao, Zhongpai Choudhuri, Anwesa Planche, Benjamin Zheng, Meng Wang, Bin Chen, Terrence Chen, Chen Wu, Ziyan Computer Vision and Pattern Recognition Temporal awareness is essential for video large language models (LLMs) to understand and reason about events within long videos, enabling applications like dense video captioning and temporal video grounding in a unified system. However, the scarcity of long videos with detailed captions and precise temporal annotations limits their temporal awareness. In this paper, we propose Seq2Time, a data-oriented training paradigm that leverages sequences of images and short video clips to enhance temporal awareness in long videos. By converting sequence positions into temporal annotations, we transform large-scale image and clip captioning datasets into sequences that mimic the temporal structure of long videos, enabling self-supervised training with abundant time-sensitive data. To enable sequence-to-time knowledge transfer, we introduce a novel time representation that unifies positional information across image sequences, clip sequences, and long videos. Experiments demonstrate the effectiveness of our method, achieving a 27.6% improvement in F1 score and 44.8% in CIDEr on the YouCook2 benchmark and a 14.7% increase in recall on the Charades-STA benchmark compared to the baseline. |
| title | Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.16932 |