Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909872062201856 |
|---|---|
| author | Shen, Yunhang Fu, Chaoyou Dong, Shaoqi Wang, Xiong Zhang, Yi-Fan Chen, Peixian Zhang, Mengdan Cao, Haoyu Li, Ke Lin, Shaohui Zheng, Xiawu Zhang, Yan Zhou, Yiyi He, Ran Shan, Caifeng Ji, Rongrong Sun, Xing |
| author_facet | Shen, Yunhang Fu, Chaoyou Dong, Shaoqi Wang, Xiong Zhang, Yi-Fan Chen, Peixian Zhang, Mengdan Cao, Haoyu Li, Ke Lin, Shaohui Zheng, Xiawu Zhang, Yan Zhou, Yiyi He, Ran Shan, Caifeng Ji, Rongrong Sun, Xing |
| contents | We introduce Long-VITA, a simple yet effective large multi-modal model for long-context visual-language understanding tasks. It is adept at concurrently processing and analyzing modalities of image, video, and text over 4K frames or 1M tokens while delivering advanced performances on short-context multi-modal tasks. We propose an effective multi-modal training schema that starts with large language models and proceeds through vision-language alignment, general knowledge learning, and two sequential stages of long-sequence fine-tuning. We further implement context-parallelism distributed inference and logits-masked language modeling head to scale Long-VITA to infinitely long inputs of images and texts during model inference. Regarding training data, Long-VITA is built on a mix of 17M samples from public datasets only and demonstrates state-of-the-art performance on various multi-modal benchmarks, compared against recent cutting-edge models with internal data. Long-VITA is fully open-source and reproducible.. By leveraging our inference designs, Long-VITA models achieve a remarkable 2x prefill speedup and 4x context length extension in a single node with 8 GPUs. We hope Long-VITA can serve as a competitive baseline and offer valuable insights for the open-source community in advancing long-context multi-modal understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_05177 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy Shen, Yunhang Fu, Chaoyou Dong, Shaoqi Wang, Xiong Zhang, Yi-Fan Chen, Peixian Zhang, Mengdan Cao, Haoyu Li, Ke Lin, Shaohui Zheng, Xiawu Zhang, Yan Zhou, Yiyi He, Ran Shan, Caifeng Ji, Rongrong Sun, Xing Computer Vision and Pattern Recognition We introduce Long-VITA, a simple yet effective large multi-modal model for long-context visual-language understanding tasks. It is adept at concurrently processing and analyzing modalities of image, video, and text over 4K frames or 1M tokens while delivering advanced performances on short-context multi-modal tasks. We propose an effective multi-modal training schema that starts with large language models and proceeds through vision-language alignment, general knowledge learning, and two sequential stages of long-sequence fine-tuning. We further implement context-parallelism distributed inference and logits-masked language modeling head to scale Long-VITA to infinitely long inputs of images and texts during model inference. Regarding training data, Long-VITA is built on a mix of 17M samples from public datasets only and demonstrates state-of-the-art performance on various multi-modal benchmarks, compared against recent cutting-edge models with internal data. Long-VITA is fully open-source and reproducible.. By leveraging our inference designs, Long-VITA models achieve a remarkable 2x prefill speedup and 4x context length extension in a single node with 8 GPUs. We hope Long-VITA can serve as a competitive baseline and offer valuable insights for the open-source community in advancing long-context multi-modal understanding. |
| title | Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.05177 |