Multimodal Evaluation of Russian-language Architectures
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908788026507264 |
|---|---|
| author | Chervyakov, Artem Isaeva, Ulyana Emelyanov, Anton Safin, Artem Tikhonova, Maria Kharitonov, Alexander Lyakh, Yulia Surovtsev, Petr Shevelev, Denis Saburov, Vildan Konovalov, Vasily Rykov, Elisei Sviridov, Ivan Miftakhova, Amina Alimova, Ilseyar Panchenko, Alexander Kapitanov, Alexander Fenogenova, Alena |
| author_facet | Chervyakov, Artem Isaeva, Ulyana Emelyanov, Anton Safin, Artem Tikhonova, Maria Kharitonov, Alexander Lyakh, Yulia Surovtsev, Petr Shevelev, Denis Saburov, Vildan Konovalov, Vasily Rykov, Elisei Sviridov, Ivan Miftakhova, Amina Alimova, Ilseyar Panchenko, Alexander Kapitanov, Alexander Fenogenova, Alena |
| contents | Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, and risks remain insufficiently understood. To address these issues, particularly in the context of the Russian language, where no multimodal benchmarks currently exist, we introduce MERA Multi, an open multimodal evaluation framework for Russian-spoken architectures. The benchmark is instruction-based and encompasses default text, image, audio, and video modalities, comprising 18 newly constructed evaluation tasks for both general-purpose models and modality-specific architectures (imageto-text, video-to-text, and audio-to-text). Our contributions include: (i) a universal taxonomy of multimodal abilities; (ii) 18 datasets created entirely from scratch with attention to Russian cultural and linguistic specificity, unified prompts, and metrics; (iii) baseline results for both closed-source and open-source models; (iv) a methodology for preventing benchmark leakage, including watermarking for private sets. While our current focus is on Russian, the proposed benchmark provides a replicable methodology for constructing multimodal benchmarks in typologically diverse languages, particularly within the Slavic language family. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_15552 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multimodal Evaluation of Russian-language Architectures Chervyakov, Artem Isaeva, Ulyana Emelyanov, Anton Safin, Artem Tikhonova, Maria Kharitonov, Alexander Lyakh, Yulia Surovtsev, Petr Shevelev, Denis Saburov, Vildan Konovalov, Vasily Rykov, Elisei Sviridov, Ivan Miftakhova, Amina Alimova, Ilseyar Panchenko, Alexander Kapitanov, Alexander Fenogenova, Alena Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, and risks remain insufficiently understood. To address these issues, particularly in the context of the Russian language, where no multimodal benchmarks currently exist, we introduce MERA Multi, an open multimodal evaluation framework for Russian-spoken architectures. The benchmark is instruction-based and encompasses default text, image, audio, and video modalities, comprising 18 newly constructed evaluation tasks for both general-purpose models and modality-specific architectures (imageto-text, video-to-text, and audio-to-text). Our contributions include: (i) a universal taxonomy of multimodal abilities; (ii) 18 datasets created entirely from scratch with attention to Russian cultural and linguistic specificity, unified prompts, and metrics; (iii) baseline results for both closed-source and open-source models; (iv) a methodology for preventing benchmark leakage, including watermarking for private sets. While our current focus is on Russian, the proposed benchmark provides a replicable methodology for constructing multimodal benchmarks in typologically diverse languages, particularly within the Slavic language family. |
| title | Multimodal Evaluation of Russian-language Architectures |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.15552 |