Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909214360731648 |
|---|---|
| author | Qian, Long Li, Juncheng Wu, Yu Ye, Yaobo Fei, Hao Chua, Tat-Seng Zhuang, Yueting Tang, Siliang |
| author_facet | Qian, Long Li, Juncheng Wu, Yu Ye, Yaobo Fei, Hao Chua, Tat-Seng Zhuang, Yueting Tang, Siliang |
| contents | Large Language Models (LLMs) demonstrate remarkable proficiency in comprehending and handling text-based tasks. Many efforts are being made to transfer these attributes to video modality, which are termed Video-LLMs. However, existing Video-LLMs can only capture the coarse-grained semantics and are unable to effectively handle tasks related to comprehension or localization of specific video segments. In light of these challenges, we propose Momentor, a Video-LLM capable of accomplishing fine-grained temporal understanding tasks. To support the training of Momentor, we design an automatic data generation engine to construct Moment-10M, a large-scale video instruction dataset with segment-level instruction data. We train Momentor on Moment-10M, enabling it to perform segment-level reasoning and localization. Zero-shot evaluations on several tasks demonstrate that Momentor excels in fine-grained temporally grounded comprehension and localization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11435 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning Qian, Long Li, Juncheng Wu, Yu Ye, Yaobo Fei, Hao Chua, Tat-Seng Zhuang, Yueting Tang, Siliang Computer Vision and Pattern Recognition Large Language Models (LLMs) demonstrate remarkable proficiency in comprehending and handling text-based tasks. Many efforts are being made to transfer these attributes to video modality, which are termed Video-LLMs. However, existing Video-LLMs can only capture the coarse-grained semantics and are unable to effectively handle tasks related to comprehension or localization of specific video segments. In light of these challenges, we propose Momentor, a Video-LLM capable of accomplishing fine-grained temporal understanding tasks. To support the training of Momentor, we design an automatic data generation engine to construct Moment-10M, a large-scale video instruction dataset with segment-level instruction data. We train Momentor on Moment-10M, enabling it to perform segment-level reasoning and localization. Zero-shot evaluations on several tasks demonstrate that Momentor excels in fine-grained temporally grounded comprehension and localization. |
| title | Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.11435 |