LongVILA: Scaling Long-Context Visual Language Models for Long Videos
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908162661023744 |
|---|---|
| author | Chen, Yukang Xue, Fuzhao Li, Dacheng Hu, Qinghao Zhu, Ligeng Li, Xiuyu Fang, Yunhao Tang, Haotian Yang, Shang Liu, Zhijian He, Ethan Yin, Hongxu Molchanov, Pavlo Kautz, Jan Fan, Linxi Zhu, Yuke Lu, Yao Han, Song |
| author_facet | Chen, Yukang Xue, Fuzhao Li, Dacheng Hu, Qinghao Zhu, Ligeng Li, Xiuyu Fang, Yunhao Tang, Haotian Yang, Shang Liu, Zhijian He, Ethan Yin, Hongxu Molchanov, Pavlo Kautz, Jan Fan, Linxi Zhu, Yuke Lu, Yao Han, Song |
| contents | Long-context capability is critical for multi-modal foundation models, especially for long video understanding. We introduce LongVILA, a full-stack solution for long-context visual-language models by co-designing the algorithm and system. For model training, we upgrade existing VLMs to support long video understanding by incorporating two additional stages, i.e., long context extension and long video supervised fine-tuning. However, training on long video is computationally and memory intensive. We introduce the long-context Multi-Modal Sequence Parallelism (MM-SP) system that efficiently parallelizes long video training and inference, enabling 2M context length training on 256 GPUs without any gradient checkpointing. LongVILA efficiently extends the number of video frames of VILA from 8 to 2048, achieving 99.8% accuracy in 6,000-frame (more than 1 million tokens) video needle-in-a-haystack. LongVILA-7B demonstrates strong accuracy on 9 popular video benchmarks, e.g. 65.1% VideoMME with subtitle. Besides, MM-SP is 2.1x - 5.7x faster than ring style sequence parallelism and 1.1x - 1.4x faster than Megatron with a hybrid context and tensor parallelism. Moreover, it seamlessly integrates with Hugging Face Transformers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10188 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LongVILA: Scaling Long-Context Visual Language Models for Long Videos Chen, Yukang Xue, Fuzhao Li, Dacheng Hu, Qinghao Zhu, Ligeng Li, Xiuyu Fang, Yunhao Tang, Haotian Yang, Shang Liu, Zhijian He, Ethan Yin, Hongxu Molchanov, Pavlo Kautz, Jan Fan, Linxi Zhu, Yuke Lu, Yao Han, Song Computer Vision and Pattern Recognition Computation and Language Long-context capability is critical for multi-modal foundation models, especially for long video understanding. We introduce LongVILA, a full-stack solution for long-context visual-language models by co-designing the algorithm and system. For model training, we upgrade existing VLMs to support long video understanding by incorporating two additional stages, i.e., long context extension and long video supervised fine-tuning. However, training on long video is computationally and memory intensive. We introduce the long-context Multi-Modal Sequence Parallelism (MM-SP) system that efficiently parallelizes long video training and inference, enabling 2M context length training on 256 GPUs without any gradient checkpointing. LongVILA efficiently extends the number of video frames of VILA from 8 to 2048, achieving 99.8% accuracy in 6,000-frame (more than 1 million tokens) video needle-in-a-haystack. LongVILA-7B demonstrates strong accuracy on 9 popular video benchmarks, e.g. 65.1% VideoMME with subtitle. Besides, MM-SP is 2.1x - 5.7x faster than ring style sequence parallelism and 1.1x - 1.4x faster than Megatron with a hybrid context and tensor parallelism. Moreover, it seamlessly integrates with Hugging Face Transformers. |
| title | LongVILA: Scaling Long-Context Visual Language Models for Long Videos |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2408.10188 |