VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909634226290688 |
|---|---|
| author | Zhang, Boqiang Li, Kehan Cheng, Zesen Hu, Zhiqiang Yuan, Yuqian Chen, Guanzheng Leng, Sicong Jiang, Yuming Zhang, Hang Li, Xin Jin, Peng Zhang, Wenqi Wang, Fan Bing, Lidong Zhao, Deli |
| author_facet | Zhang, Boqiang Li, Kehan Cheng, Zesen Hu, Zhiqiang Yuan, Yuqian Chen, Guanzheng Leng, Sicong Jiang, Yuming Zhang, Hang Li, Xin Jin, Peng Zhang, Wenqi Wang, Fan Bing, Lidong Zhao, Deli |
| contents | In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation model for image and video understanding. The core design philosophy of VideoLLaMA3 is vision-centric. The meaning of "vision-centric" is two-fold: the vision-centric training paradigm and vision-centric framework design. The key insight of our vision-centric training paradigm is that high-quality image-text data is crucial for both image and video understanding. Instead of preparing massive video-text datasets, we focus on constructing large-scale and high-quality image-text datasets. VideoLLaMA3 has four training stages: 1) Vision Encoder Adaptation, which enables vision encoder to accept images of variable resolutions as input; 2) Vision-Language Alignment, which jointly tunes the vision encoder, projector, and LLM with large-scale image-text data covering multiple types (including scene images, documents, charts) as well as text-only data. 3) Multi-task Fine-tuning, which incorporates image-text SFT data for downstream tasks and video-text data to establish a foundation for video understanding. 4) Video-centric Fine-tuning, which further improves the model's capability in video understanding. As for the framework design, to better capture fine-grained details in images, the pretrained vision encoder is adapted to encode images of varying sizes into vision tokens with corresponding numbers, rather than a fixed number of tokens. For video inputs, we reduce the number of vision tokens according to their similarity so that the representation of videos will be more precise and compact. Benefit from vision-centric designs, VideoLLaMA3 achieves compelling performances in both image and video understanding benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_13106 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding Zhang, Boqiang Li, Kehan Cheng, Zesen Hu, Zhiqiang Yuan, Yuqian Chen, Guanzheng Leng, Sicong Jiang, Yuming Zhang, Hang Li, Xin Jin, Peng Zhang, Wenqi Wang, Fan Bing, Lidong Zhao, Deli Computer Vision and Pattern Recognition In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation model for image and video understanding. The core design philosophy of VideoLLaMA3 is vision-centric. The meaning of "vision-centric" is two-fold: the vision-centric training paradigm and vision-centric framework design. The key insight of our vision-centric training paradigm is that high-quality image-text data is crucial for both image and video understanding. Instead of preparing massive video-text datasets, we focus on constructing large-scale and high-quality image-text datasets. VideoLLaMA3 has four training stages: 1) Vision Encoder Adaptation, which enables vision encoder to accept images of variable resolutions as input; 2) Vision-Language Alignment, which jointly tunes the vision encoder, projector, and LLM with large-scale image-text data covering multiple types (including scene images, documents, charts) as well as text-only data. 3) Multi-task Fine-tuning, which incorporates image-text SFT data for downstream tasks and video-text data to establish a foundation for video understanding. 4) Video-centric Fine-tuning, which further improves the model's capability in video understanding. As for the framework design, to better capture fine-grained details in images, the pretrained vision encoder is adapted to encode images of varying sizes into vision tokens with corresponding numbers, rather than a fixed number of tokens. For video inputs, we reduce the number of vision tokens according to their similarity so that the representation of videos will be more precise and compact. Benefit from vision-centric designs, VideoLLaMA3 achieves compelling performances in both image and video understanding benchmarks. |
| title | VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.13106 |