Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929600341213184 |
|---|---|
| author | Paniv, Yurii Kiulian, Artur Chaplynskyi, Dmytro Khandoga, Mykola Polishko, Anton Bas, Tetiana Gabrielli, Guillermo |
| author_facet | Paniv, Yurii Kiulian, Artur Chaplynskyi, Dmytro Khandoga, Mykola Polishko, Anton Bas, Tetiana Gabrielli, Guillermo |
| contents | While the evaluation of multimodal English-centric models is an active area of research with numerous benchmarks, there is a profound lack of benchmarks or evaluation suites for low- and mid-resource languages. We introduce ZNO-Vision, a comprehensive multimodal Ukrainian-centric benchmark derived from standardized university entrance examination (ZNO). The benchmark consists of over 4,300 expert-crafted questions spanning 12 academic disciplines, including mathematics, physics, chemistry, and humanities. We evaluated the performance of both open-source models and API providers, finding that only a handful of models performed above baseline. Alongside the new benchmark, we performed the first evaluation study of multimodal text generation for the Ukrainian language: we measured caption generation quality on the Multi30K-UK dataset, translated the VQA benchmark into Ukrainian, and measured performance degradation relative to original English versions. Lastly, we tested a few models from a cultural perspective on knowledge of national cuisine. We believe our work will advance multimodal generation capabilities for the Ukrainian language and our approach could be useful for other low-resource languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_14647 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains Paniv, Yurii Kiulian, Artur Chaplynskyi, Dmytro Khandoga, Mykola Polishko, Anton Bas, Tetiana Gabrielli, Guillermo Computation and Language While the evaluation of multimodal English-centric models is an active area of research with numerous benchmarks, there is a profound lack of benchmarks or evaluation suites for low- and mid-resource languages. We introduce ZNO-Vision, a comprehensive multimodal Ukrainian-centric benchmark derived from standardized university entrance examination (ZNO). The benchmark consists of over 4,300 expert-crafted questions spanning 12 academic disciplines, including mathematics, physics, chemistry, and humanities. We evaluated the performance of both open-source models and API providers, finding that only a handful of models performed above baseline. Alongside the new benchmark, we performed the first evaluation study of multimodal text generation for the Ukrainian language: we measured caption generation quality on the Multi30K-UK dataset, translated the VQA benchmark into Ukrainian, and measured performance degradation relative to original English versions. Lastly, we tested a few models from a cultural perspective on knowledge of national cuisine. We believe our work will advance multimodal generation capabilities for the Ukrainian language and our approach could be useful for other low-resource languages. |
| title | Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.14647 |