Long Context Transfer from Language to Vision
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911937511555072 |
|---|---|
| author | Zhang, Peiyuan Zhang, Kaichen Li, Bo Zeng, Guangtao Yang, Jingkang Zhang, Yuanhan Wang, Ziyue Tan, Haoran Li, Chunyuan Liu, Ziwei |
| author_facet | Zhang, Peiyuan Zhang, Kaichen Li, Bo Zeng, Guangtao Yang, Jingkang Zhang, Yuanhan Wang, Ziyue Tan, Haoran Li, Chunyuan Liu, Ziwei |
| contents | Video sequences offer valuable temporal information, but existing large multimodal models (LMMs) fall short in understanding extremely long videos. Many works address this by reducing the number of visual tokens using visual resamplers. Alternatively, in this paper, we approach this problem from the perspective of the language model. By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training. We call this phenomenon long context transfer and carefully ablate its properties. To effectively measure LMMs' ability to generalize to long contexts in the vision modality, we develop V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark inspired by the language model's NIAH test. Our proposed Long Video Assistant (LongVA) can process 2000 frames or over 200K visual tokens without additional complexities. With its extended context length, LongVA achieves state-of-the-art performance on Video-MME among 7B-scale models by densely sampling more input frames. Our work is open-sourced at https://github.com/EvolvingLMMs-Lab/LongVA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16852 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Long Context Transfer from Language to Vision Zhang, Peiyuan Zhang, Kaichen Li, Bo Zeng, Guangtao Yang, Jingkang Zhang, Yuanhan Wang, Ziyue Tan, Haoran Li, Chunyuan Liu, Ziwei Computer Vision and Pattern Recognition Video sequences offer valuable temporal information, but existing large multimodal models (LMMs) fall short in understanding extremely long videos. Many works address this by reducing the number of visual tokens using visual resamplers. Alternatively, in this paper, we approach this problem from the perspective of the language model. By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training. We call this phenomenon long context transfer and carefully ablate its properties. To effectively measure LMMs' ability to generalize to long contexts in the vision modality, we develop V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark inspired by the language model's NIAH test. Our proposed Long Video Assistant (LongVA) can process 2000 frames or over 200K visual tokens without additional complexities. With its extended context length, LongVA achieves state-of-the-art performance on Video-MME among 7B-scale models by densely sampling more input frames. Our work is open-sourced at https://github.com/EvolvingLMMs-Lab/LongVA. |
| title | Long Context Transfer from Language to Vision |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.16852 |