MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916073617489920 |
|---|---|
| author | Wei, Yujie Han, Yujin Chen, Zhekai Li, Yongming Jiang, Kaixun Liu, Zhihang Li, Quanhao Qing, Zhiwu Wang, Xiang Xing, Zhen Chu, Ruihang Hong, Lingyi He, Yefei Zhou, Junjie Yu, Junqiu Shi, Yang Zou, Difan Zhu, Kai Zhang, Shiwei Zhang, Yingya Liu, Yu Liu, Xihui Shan, Hongming |
| author_facet | Wei, Yujie Han, Yujin Chen, Zhekai Li, Yongming Jiang, Kaixun Liu, Zhihang Li, Quanhao Qing, Zhiwu Wang, Xiang Xing, Zhen Chu, Ruihang Hong, Lingyi He, Yefei Zhou, Junjie Yu, Junqiu Shi, Yang Zou, Difan Zhu, Kai Zhang, Shiwei Zhang, Yingya Liu, Yu Liu, Xihui Shan, Hongming |
| contents | Video generation is rapidly evolving from single-shot synthesis to complex multi-shot audio-video (MSAV) narratives to meet real-world demands. However, evaluating such frontier models remains a fundamental challenge. Existing benchmarks are limited in scope and data diversity, and rely on rigid evaluation pipelines, preventing systematic and reliable assessment of modern MSAV models. To bridge these gaps, we introduce MSAVBench, the first comprehensive benchmark and adaptive hybrid evaluation framework for multi-shot audio-video generation. Our benchmark spans four key dimensions, video, audio, shot, and reference, covering diverse task settings, varying shot counts of up to 15, and challenging non-realistic scenarios. Our evaluation framework improves robustness through an adaptive self-correction mechanism for shot segmentation, instance-wise rubrics for subjective metrics, and tool-grounded evidence extraction for complex judgments. Furthermore, MSAVBench achieves high alignment with human judgments, reaching a Spearman rank correlation of 91.5%. Our systematic evaluation of 19 state-of-the-art closed- and open-source models shows that current systems still struggle with director-level control and fine-grained audio-visual synchronization, while modular or agentic generation pipelines offer a promising path toward narrowing the gap between open- and closed-source models. The benchmark data and evaluation code are publicly available at https://github.com/ali-vilab/MSAVBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_20183 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation Wei, Yujie Han, Yujin Chen, Zhekai Li, Yongming Jiang, Kaixun Liu, Zhihang Li, Quanhao Qing, Zhiwu Wang, Xiang Xing, Zhen Chu, Ruihang Hong, Lingyi He, Yefei Zhou, Junjie Yu, Junqiu Shi, Yang Zou, Difan Zhu, Kai Zhang, Shiwei Zhang, Yingya Liu, Yu Liu, Xihui Shan, Hongming Computer Vision and Pattern Recognition Video generation is rapidly evolving from single-shot synthesis to complex multi-shot audio-video (MSAV) narratives to meet real-world demands. However, evaluating such frontier models remains a fundamental challenge. Existing benchmarks are limited in scope and data diversity, and rely on rigid evaluation pipelines, preventing systematic and reliable assessment of modern MSAV models. To bridge these gaps, we introduce MSAVBench, the first comprehensive benchmark and adaptive hybrid evaluation framework for multi-shot audio-video generation. Our benchmark spans four key dimensions, video, audio, shot, and reference, covering diverse task settings, varying shot counts of up to 15, and challenging non-realistic scenarios. Our evaluation framework improves robustness through an adaptive self-correction mechanism for shot segmentation, instance-wise rubrics for subjective metrics, and tool-grounded evidence extraction for complex judgments. Furthermore, MSAVBench achieves high alignment with human judgments, reaching a Spearman rank correlation of 91.5%. Our systematic evaluation of 19 state-of-the-art closed- and open-source models shows that current systems still struggle with director-level control and fine-grained audio-visual synchronization, while modular or agentic generation pipelines offer a promising path toward narrowing the gap between open- and closed-source models. The benchmark data and evaluation code are publicly available at https://github.com/ali-vilab/MSAVBench. |
| title | MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.20183 |