Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915126281502720 |
|---|---|
| author | Li, Yun Liu, Zhe Kong, Yajing Li, Guangrui Zhang, Jiyuan Bian, Chao Liu, Feng Yao, Lina Sun, Zhenbang |
| author_facet | Li, Yun Liu, Zhe Kong, Yajing Li, Guangrui Zhang, Jiyuan Bian, Chao Liu, Feng Yao, Lina Sun, Zhenbang |
| contents | Applying Multimodal Large Language Models (MLLMs) to video understanding presents significant challenges due to the need to model temporal relations across frames. Existing approaches adopt either implicit temporal modeling, relying solely on the LLM decoder, or explicit temporal modeling, employing auxiliary temporal encoders. To investigate this debate between the two paradigms, we propose the Stackable Temporal Encoder (STE). STE enables flexible explicit temporal modeling with adjustable temporal receptive fields and token compression ratios. Using STE, we systematically compare implicit and explicit temporal modeling across dimensions such as overall performance, token compression effectiveness, and temporal-specific understanding. We also explore STE's design considerations and broader impacts as a plug-in module and in image modalities. Our findings emphasize the critical role of explicit temporal modeling, providing actionable insights to advance video MLLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16786 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding Li, Yun Liu, Zhe Kong, Yajing Li, Guangrui Zhang, Jiyuan Bian, Chao Liu, Feng Yao, Lina Sun, Zhenbang Computer Vision and Pattern Recognition Computation and Language Applying Multimodal Large Language Models (MLLMs) to video understanding presents significant challenges due to the need to model temporal relations across frames. Existing approaches adopt either implicit temporal modeling, relying solely on the LLM decoder, or explicit temporal modeling, employing auxiliary temporal encoders. To investigate this debate between the two paradigms, we propose the Stackable Temporal Encoder (STE). STE enables flexible explicit temporal modeling with adjustable temporal receptive fields and token compression ratios. Using STE, we systematically compare implicit and explicit temporal modeling across dimensions such as overall performance, token compression effectiveness, and temporal-specific understanding. We also explore STE's design considerations and broader impacts as a plug-in module and in image modalities. Our findings emphasize the critical role of explicit temporal modeling, providing actionable insights to advance video MLLMs. |
| title | Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2501.16786 |