KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918451775275008 |
|---|---|
| author | Lee, Nahyun Son, Guijin Ko, Hyunwoo Kim, Chanyoung An, JunYoung Han, Kyubeen Kwak, Il-Youp |
| author_facet | Lee, Nahyun Son, Guijin Ko, Hyunwoo Kim, Chanyoung An, JunYoung Han, Kyubeen Kwak, Il-Youp |
| contents | We introduce KMMMU, a native Korean benchmark for evaluating multimodal understanding in Korean cultural and institutional settings. KMMMU contains 3,466 questions from exams natively written in Korean, covering nine disciplines and nine visual modality categories, along with a 300-item Korean-specific subset and a hard subset of 627 questions. Unlike translated or English-centric benchmarks, KMMMU targets information-dense problems shaped by local conventions, official standards, and discipline-specific visual formats. Experiments show that the strongest open-source model reaches only 42.05% accuracy on the full set, while the best proprietary model achieves 52.42% on the hard subset. Performance varies across disciplines, with some disciplines emerging as bottlenecks, and Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that these failures stem less from insufficient reasoning depth than from weak convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards understanding. KMMMU provides a testbed for multimodal evaluation beyond English-centric benchmarks and for developing more reliable systems for expert real-world tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_13058 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context Lee, Nahyun Son, Guijin Ko, Hyunwoo Kim, Chanyoung An, JunYoung Han, Kyubeen Kwak, Il-Youp Computation and Language Machine Learning Multimedia We introduce KMMMU, a native Korean benchmark for evaluating multimodal understanding in Korean cultural and institutional settings. KMMMU contains 3,466 questions from exams natively written in Korean, covering nine disciplines and nine visual modality categories, along with a 300-item Korean-specific subset and a hard subset of 627 questions. Unlike translated or English-centric benchmarks, KMMMU targets information-dense problems shaped by local conventions, official standards, and discipline-specific visual formats. Experiments show that the strongest open-source model reaches only 42.05% accuracy on the full set, while the best proprietary model achieves 52.42% on the hard subset. Performance varies across disciplines, with some disciplines emerging as bottlenecks, and Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that these failures stem less from insufficient reasoning depth than from weak convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards understanding. KMMMU provides a testbed for multimodal evaluation beyond English-centric benchmarks and for developing more reliable systems for expert real-world tasks. |
| title | KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context |
| topic | Computation and Language Machine Learning Multimedia |
| url | https://arxiv.org/abs/2604.13058 |