OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917971612401664 |
|---|---|
| author | Wang, Yuxuan Wang, Yueqian Chen, Bo Wu, Tong Zhao, Dongyan Zheng, Zilong |
| author_facet | Wang, Yuxuan Wang, Yueqian Chen, Bo Wu, Tong Zhao, Dongyan Zheng, Zilong |
| contents | The rapid advancement of multi-modal language models (MLLMs) like GPT-4o has propelled the development of Omni language models, designed to process and proactively respond to continuous streams of multi-modal data. Despite their potential, evaluating their real-world interactive capabilities in streaming video contexts remains a formidable challenge. In this work, we introduce OmniMMI, a comprehensive multi-modal interaction benchmark tailored for OmniLLMs in streaming video contexts. OmniMMI encompasses over 1,121 videos and 2,290 questions, addressing two critical yet underexplored challenges in existing video benchmarks: streaming video understanding and proactive reasoning, across six distinct subtasks. Moreover, we propose a novel framework, Multi-modal Multiplexing Modeling (M4), designed to enable an inference-efficient streaming model that can see, listen while generating. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_22952 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts Wang, Yuxuan Wang, Yueqian Chen, Bo Wu, Tong Zhao, Dongyan Zheng, Zilong Computer Vision and Pattern Recognition The rapid advancement of multi-modal language models (MLLMs) like GPT-4o has propelled the development of Omni language models, designed to process and proactively respond to continuous streams of multi-modal data. Despite their potential, evaluating their real-world interactive capabilities in streaming video contexts remains a formidable challenge. In this work, we introduce OmniMMI, a comprehensive multi-modal interaction benchmark tailored for OmniLLMs in streaming video contexts. OmniMMI encompasses over 1,121 videos and 2,290 questions, addressing two critical yet underexplored challenges in existing video benchmarks: streaming video understanding and proactive reasoning, across six distinct subtasks. Moreover, we propose a novel framework, Multi-modal Multiplexing Modeling (M4), designed to enable an inference-efficient streaming model that can see, listen while generating. |
| title | OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.22952 |