Streaming Video Question-Answering with In-context Video KV-Cache Retrieval
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917942527000576 |
|---|---|
| author | Di, Shangzhe Yu, Zhelun Zhang, Guanghao Li, Haoyuan Zhong, Tao Cheng, Hao Li, Bolin He, Wanggui Shu, Fangxun Jiang, Hao |
| author_facet | Di, Shangzhe Yu, Zhelun Zhang, Guanghao Li, Haoyuan Zhong, Tao Cheng, Hao Li, Bolin He, Wanggui Shu, Fangxun Jiang, Hao |
| contents | We propose ReKV, a novel training-free approach that enables efficient streaming video question-answering (StreamingVQA), by seamlessly integrating with existing Video Large Language Models (Video-LLMs). Traditional VideoQA systems struggle with long videos, as they must process entire videos before responding to queries, and repeat this process for each new question. In contrast, our approach analyzes long videos in a streaming manner, allowing for prompt responses as soon as user queries are received. Building on a common Video-LLM, we first incorporate a sliding-window attention mechanism, ensuring that input frames attend to a limited number of preceding frames, thereby reducing computational overhead. To prevent information loss, we store processed video key-value caches (KV-Caches) in RAM and disk, reloading them into GPU memory as needed. Additionally, we introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMs to retrieve only query-relevant KV-Caches, ensuring both efficiency and accuracy in question answering. ReKV enables the separation of video encoding and question-answering across different processes and GPUs, significantly enhancing the efficiency of StreamingVQA. Through comprehensive experimentation, we validate the efficacy and practicality of our approach, which significantly boosts efficiency and enhances applicability over existing VideoQA models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_00540 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Streaming Video Question-Answering with In-context Video KV-Cache Retrieval Di, Shangzhe Yu, Zhelun Zhang, Guanghao Li, Haoyuan Zhong, Tao Cheng, Hao Li, Bolin He, Wanggui Shu, Fangxun Jiang, Hao Computer Vision and Pattern Recognition We propose ReKV, a novel training-free approach that enables efficient streaming video question-answering (StreamingVQA), by seamlessly integrating with existing Video Large Language Models (Video-LLMs). Traditional VideoQA systems struggle with long videos, as they must process entire videos before responding to queries, and repeat this process for each new question. In contrast, our approach analyzes long videos in a streaming manner, allowing for prompt responses as soon as user queries are received. Building on a common Video-LLM, we first incorporate a sliding-window attention mechanism, ensuring that input frames attend to a limited number of preceding frames, thereby reducing computational overhead. To prevent information loss, we store processed video key-value caches (KV-Caches) in RAM and disk, reloading them into GPU memory as needed. Additionally, we introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMs to retrieve only query-relevant KV-Caches, ensuring both efficiency and accuracy in question answering. ReKV enables the separation of video encoding and question-answering across different processes and GPUs, significantly enhancing the efficiency of StreamingVQA. Through comprehensive experimentation, we validate the efficacy and practicality of our approach, which significantly boosts efficiency and enhances applicability over existing VideoQA models. |
| title | Streaming Video Question-Answering with In-context Video KV-Cache Retrieval |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.00540 |