Mavors: Multi-granularity Video Representation for Multimodal Large Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911288953667584 |
|---|---|
| author | Shi, Yang Liu, Jiaheng Guan, Yushuo Wu, Zhenhua Zhang, Yuanxing Wang, Zihao Lin, Weihong Hua, Jingyun Wang, Zekun Chen, Xinlong Zeng, Bohan Zhang, Wentao Zhang, Fuzheng Yang, Wenjing Zhang, Di |
| author_facet | Shi, Yang Liu, Jiaheng Guan, Yushuo Wu, Zhenhua Zhang, Yuanxing Wang, Zihao Lin, Weihong Hua, Jingyun Wang, Zekun Chen, Xinlong Zeng, Bohan Zhang, Wentao Zhang, Fuzheng Yang, Wenjing Zhang, Di |
| contents | Long-context video understanding in multimodal large language models (MLLMs) faces a critical challenge: balancing computational efficiency with the retention of fine-grained spatio-temporal patterns. Existing approaches (e.g., sparse sampling, dense sampling with low resolution, and token compression) suffer from significant information loss in temporal dynamics, spatial details, or subtle interactions, particularly in videos with complex motion or varying resolutions. To address this, we propose $\mathbf{Mavors}$, a novel framework that introduces $\mathbf{M}$ulti-gr$\mathbf{a}$nularity $\mathbf{v}$ide$\mathbf{o}$ $\mathbf{r}$epre$\mathbf{s}$entation for holistic long-video modeling. Specifically, Mavors directly encodes raw video content into latent representations through two core components: 1) an Intra-chunk Vision Encoder (IVE) that preserves high-resolution spatial features via 3D convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator (IFA) that establishes temporal coherence across chunks using transformer-based dependency modeling with chunk-level rotary position encodings. Moreover, the framework unifies image and video understanding by treating images as single-frame videos via sub-image decomposition. Experiments across diverse benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity and temporal continuity, significantly outperforming existing methods in tasks requiring fine-grained spatio-temporal reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_10068 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mavors: Multi-granularity Video Representation for Multimodal Large Language Model Shi, Yang Liu, Jiaheng Guan, Yushuo Wu, Zhenhua Zhang, Yuanxing Wang, Zihao Lin, Weihong Hua, Jingyun Wang, Zekun Chen, Xinlong Zeng, Bohan Zhang, Wentao Zhang, Fuzheng Yang, Wenjing Zhang, Di Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Long-context video understanding in multimodal large language models (MLLMs) faces a critical challenge: balancing computational efficiency with the retention of fine-grained spatio-temporal patterns. Existing approaches (e.g., sparse sampling, dense sampling with low resolution, and token compression) suffer from significant information loss in temporal dynamics, spatial details, or subtle interactions, particularly in videos with complex motion or varying resolutions. To address this, we propose $\mathbf{Mavors}$, a novel framework that introduces $\mathbf{M}$ulti-gr$\mathbf{a}$nularity $\mathbf{v}$ide$\mathbf{o}$ $\mathbf{r}$epre$\mathbf{s}$entation for holistic long-video modeling. Specifically, Mavors directly encodes raw video content into latent representations through two core components: 1) an Intra-chunk Vision Encoder (IVE) that preserves high-resolution spatial features via 3D convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator (IFA) that establishes temporal coherence across chunks using transformer-based dependency modeling with chunk-level rotary position encodings. Moreover, the framework unifies image and video understanding by treating images as single-frame videos via sub-image decomposition. Experiments across diverse benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity and temporal continuity, significantly outperforming existing methods in tasks requiring fine-grained spatio-temporal reasoning. |
| title | Mavors: Multi-granularity Video Representation for Multimodal Large Language Model |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2504.10068 |