Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916717732560896 |
|---|---|
| author | Song, Enxin Chai, Wenhao Xu, Weili Xie, Jianwen Liu, Yuxuan Wang, Gaoang |
| author_facet | Song, Enxin Chai, Wenhao Xu, Weili Xie, Jianwen Liu, Yuxuan Wang, Gaoang |
| contents | Recent advancements in language multimodal models (LMMs) for video have demonstrated their potential for understanding video content, yet the task of comprehending multi-discipline lectures remains largely unexplored. We introduce Video-MMLU, a massive benchmark designed to evaluate the capabilities of LMMs in understanding Multi-Discipline Lectures. We evaluate over 90 open-source and proprietary models, ranging from 0.5B to 40B parameters. Our results highlight the limitations of current models in addressing the cognitive challenges presented by these lectures, especially in tasks requiring both perception and reasoning. Additionally, we explore how the number of visual tokens and the large language models influence performance, offering insights into the interplay between multimodal perception and reasoning in lecture comprehension. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14693 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark Song, Enxin Chai, Wenhao Xu, Weili Xie, Jianwen Liu, Yuxuan Wang, Gaoang Computer Vision and Pattern Recognition Artificial Intelligence Recent advancements in language multimodal models (LMMs) for video have demonstrated their potential for understanding video content, yet the task of comprehending multi-discipline lectures remains largely unexplored. We introduce Video-MMLU, a massive benchmark designed to evaluate the capabilities of LMMs in understanding Multi-Discipline Lectures. We evaluate over 90 open-source and proprietary models, ranging from 0.5B to 40B parameters. Our results highlight the limitations of current models in addressing the cognitive challenges presented by these lectures, especially in tasks requiring both perception and reasoning. Additionally, we explore how the number of visual tokens and the large language models influence performance, offering insights into the interplay between multimodal perception and reasoning in lecture comprehension. |
| title | Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2504.14693 |