Hierarchical Memory for Long Video QA
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917869475856384 |
|---|---|
| author | Wang, Yiqin Zhang, Haoji Tang, Yansong Liu, Yong Feng, Jiashi Dai, Jifeng Jin, Xiaojie |
| author_facet | Wang, Yiqin Zhang, Haoji Tang, Yansong Liu, Yong Feng, Jiashi Dai, Jifeng Jin, Xiaojie |
| contents | This paper describes our champion solution to the LOVEU Challenge @ CVPR'24, Track 1 (Long Video VQA). Processing long sequences of visual tokens is computationally expensive and memory-intensive, making long video question-answering a challenging task. The key is to compress visual tokens effectively, reducing memory footprint and decoding latency, while preserving the essential information for accurate question-answering. We adopt a hierarchical memory mechanism named STAR Memory, proposed in Flash-VStream, that is capable of processing long videos with limited GPU memory (VRAM). We further utilize the video and audio data of MovieChat-1K training set to fine-tune the pretrained weight released by Flash-VStream, achieving 1st place in the challenge. Code is available at project homepage https://invinciblewyq.github.io/vstream-page . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00603 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hierarchical Memory for Long Video QA Wang, Yiqin Zhang, Haoji Tang, Yansong Liu, Yong Feng, Jiashi Dai, Jifeng Jin, Xiaojie Computer Vision and Pattern Recognition This paper describes our champion solution to the LOVEU Challenge @ CVPR'24, Track 1 (Long Video VQA). Processing long sequences of visual tokens is computationally expensive and memory-intensive, making long video question-answering a challenging task. The key is to compress visual tokens effectively, reducing memory footprint and decoding latency, while preserving the essential information for accurate question-answering. We adopt a hierarchical memory mechanism named STAR Memory, proposed in Flash-VStream, that is capable of processing long videos with limited GPU memory (VRAM). We further utilize the video and audio data of MovieChat-1K training set to fine-tune the pretrained weight released by Flash-VStream, achieving 1st place in the challenge. Code is available at project homepage https://invinciblewyq.github.io/vstream-page . |
| title | Hierarchical Memory for Long Video QA |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.00603 |