RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914256435281920 |
|---|---|
| author | Xun, Shuhang Tao, Sicheng Li, Jungang Shi, Yibo Lin, Zhixin Zhu, Zhanhui Yan, Yibo Li, Hanqian Zhang, Linghao Wang, Shikang Liu, Yixin Zhang, Hanbo Ma, Ying Hu, Xuming |
| author_facet | Xun, Shuhang Tao, Sicheng Li, Jungang Shi, Yibo Lin, Zhixin Zhu, Zhanhui Yan, Yibo Li, Hanqian Zhang, Linghao Wang, Shikang Liu, Yixin Zhang, Hanbo Ma, Ying Hu, Xuming |
| contents | Multimodal Large Language Models (MLLMs) have made rapid progress in perception, understanding, and reasoning, yet existing benchmarks fall short in evaluating these abilities under continuous and dynamic real-world video streams. Such settings require models to maintain coherent understanding and reasoning as visual scenes evolve over time. **We introduce RTV-Bench, a fine-grained benchmark for real-time video analysis with MLLMs**. It is built upon three key principles: multi-timestamp question answering, hierarchical question structures spanning perception and reasoning, and multi-dimensional evaluation of continuous perception, understanding, and reasoning. RTV-Bench comprises 552 diverse videos and 4,608 carefully curated QA pairs covering a wide range of dynamic scenarios. We evaluate a broad range of state-of-the-art MLLMs, including proprietary, open-source offline, and open-source real-time models. Our results show that real-time models generally outperform offline counterparts but still lag behind leading proprietary systems. While scaling model capacity generally yields performance gains, simply increasing the density of sampled input frames does not consistently translate into improved results. These observations suggest inherent limitations in current architectures when handling long-horizon video streams, underscoring the need for models explicitly designed for streaming video processing and analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_02064 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video Xun, Shuhang Tao, Sicheng Li, Jungang Shi, Yibo Lin, Zhixin Zhu, Zhanhui Yan, Yibo Li, Hanqian Zhang, Linghao Wang, Shikang Liu, Yixin Zhang, Hanbo Ma, Ying Hu, Xuming Computer Vision and Pattern Recognition Multimodal Large Language Models (MLLMs) have made rapid progress in perception, understanding, and reasoning, yet existing benchmarks fall short in evaluating these abilities under continuous and dynamic real-world video streams. Such settings require models to maintain coherent understanding and reasoning as visual scenes evolve over time. **We introduce RTV-Bench, a fine-grained benchmark for real-time video analysis with MLLMs**. It is built upon three key principles: multi-timestamp question answering, hierarchical question structures spanning perception and reasoning, and multi-dimensional evaluation of continuous perception, understanding, and reasoning. RTV-Bench comprises 552 diverse videos and 4,608 carefully curated QA pairs covering a wide range of dynamic scenarios. We evaluate a broad range of state-of-the-art MLLMs, including proprietary, open-source offline, and open-source real-time models. Our results show that real-time models generally outperform offline counterparts but still lag behind leading proprietary systems. While scaling model capacity generally yields performance gains, simply increasing the density of sampled input frames does not consistently translate into improved results. These observations suggest inherent limitations in current architectures when handling long-horizon video streams, underscoring the need for models explicitly designed for streaming video processing and analysis. |
| title | RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.02064 |