MERA: A Comprehensive LLM Evaluation in Russian
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913455573827584 |
|---|---|
| author | Fenogenova, Alena Chervyakov, Artem Martynov, Nikita Kozlova, Anastasia Tikhonova, Maria Akhmetgareeva, Albina Emelyanov, Anton Shevelev, Denis Lebedev, Pavel Sinev, Leonid Isaeva, Ulyana Kolomeytseva, Katerina Moskovskiy, Daniil Goncharova, Elizaveta Savushkin, Nikita Mikhailova, Polina Dimitrov, Denis Panchenko, Alexander Markov, Sergei |
| author_facet | Fenogenova, Alena Chervyakov, Artem Martynov, Nikita Kozlova, Anastasia Tikhonova, Maria Akhmetgareeva, Albina Emelyanov, Anton Shevelev, Denis Lebedev, Pavel Sinev, Leonid Isaeva, Ulyana Kolomeytseva, Katerina Moskovskiy, Daniil Goncharova, Elizaveta Savushkin, Nikita Mikhailova, Polina Dimitrov, Denis Panchenko, Alexander Markov, Sergei |
| contents | Over the past few years, one of the most notable advancements in AI research has been in foundation models (FMs), headlined by the rise of language models (LMs). As the models' size increases, LMs demonstrate enhancements in measurable aspects and the development of new qualitative features. However, despite researchers' attention and the rapid growth in LM application, the capabilities, limitations, and associated risks still need to be better understood. To address these issues, we introduce an open Multimodal Evaluation of Russian-language Architectures (MERA), a new instruction benchmark for evaluating foundation models oriented towards the Russian language. The benchmark encompasses 21 evaluation tasks for generative models in 11 skill domains and is designed as a black-box test to ensure the exclusion of data leakage. The paper introduces a methodology to evaluate FMs and LMs in zero- and few-shot fixed instruction settings that can be extended to other modalities. We propose an evaluation methodology, an open-source code base for the MERA assessment, and a leaderboard with a submission system. We evaluate open LMs as baselines and find that they are still far behind the human level. We publicly release MERA to guide forthcoming research, anticipate groundbreaking model features, standardize the evaluation procedure, and address potential societal drawbacks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_04531 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MERA: A Comprehensive LLM Evaluation in Russian Fenogenova, Alena Chervyakov, Artem Martynov, Nikita Kozlova, Anastasia Tikhonova, Maria Akhmetgareeva, Albina Emelyanov, Anton Shevelev, Denis Lebedev, Pavel Sinev, Leonid Isaeva, Ulyana Kolomeytseva, Katerina Moskovskiy, Daniil Goncharova, Elizaveta Savushkin, Nikita Mikhailova, Polina Dimitrov, Denis Panchenko, Alexander Markov, Sergei Computation and Language Artificial Intelligence Over the past few years, one of the most notable advancements in AI research has been in foundation models (FMs), headlined by the rise of language models (LMs). As the models' size increases, LMs demonstrate enhancements in measurable aspects and the development of new qualitative features. However, despite researchers' attention and the rapid growth in LM application, the capabilities, limitations, and associated risks still need to be better understood. To address these issues, we introduce an open Multimodal Evaluation of Russian-language Architectures (MERA), a new instruction benchmark for evaluating foundation models oriented towards the Russian language. The benchmark encompasses 21 evaluation tasks for generative models in 11 skill domains and is designed as a black-box test to ensure the exclusion of data leakage. The paper introduces a methodology to evaluate FMs and LMs in zero- and few-shot fixed instruction settings that can be extended to other modalities. We propose an evaluation methodology, an open-source code base for the MERA assessment, and a leaderboard with a submission system. We evaluate open LMs as baselines and find that they are still far behind the human level. We publicly release MERA to guide forthcoming research, anticipate groundbreaking model features, standardize the evaluation procedure, and address potential societal drawbacks. |
| title | MERA: A Comprehensive LLM Evaluation in Russian |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2401.04531 |