Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2412.17933 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909619279888384 |
|---|---|
| author | Fajcik, Martin Docekal, Martin Dolezal, Jan Ondrej, Karel Beneš, Karel Kapsa, Jan Smrz, Pavel Polok, Alexander Hradis, Michal Neverilova, Zuzana Horak, Ales Sabol, Radoslav Stefanik, Michal Jirkovsky, Adam Adamczyk, David Hyner, Petr Hula, Jan Kydlicek, Hynek |
| author_facet | Fajcik, Martin Docekal, Martin Dolezal, Jan Ondrej, Karel Beneš, Karel Kapsa, Jan Smrz, Pavel Polok, Alexander Hradis, Michal Neverilova, Zuzana Horak, Ales Sabol, Radoslav Stefanik, Michal Jirkovsky, Adam Adamczyk, David Hyner, Petr Hula, Jan Kydlicek, Hynek |
| contents | We present BenCzechMark (BCM), the first comprehensive Czech language benchmark designed for large language models, offering diverse tasks, multiple task formats, and multiple evaluation metrics. Its duel scoring system is grounded in statistical significance theory and uses aggregation across tasks inspired by social preference theory. Our benchmark encompasses 50 challenging tasks, with corresponding test datasets, primarily in native Czech, with 14 newly collected ones. These tasks span 8 categories and cover diverse domains, including historical Czech news, essays from pupils or language learners, and spoken word. Furthermore, we collect and clean BUT-Large Czech Collection, the largest publicly available clean Czech language corpus, and use it for (i) contamination analysis and (ii) continuous pretraining of the first Czech-centric 7B language model with Czech-specific tokenization. We use our model as a baseline for comparison with publicly available multilingual models. Lastly, we release and maintain a leaderboard with existing 50 model submissions, where new model submissions can be made at https://huggingface.co/spaces/CZLC/BenCzechMark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_17933 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism Fajcik, Martin Docekal, Martin Dolezal, Jan Ondrej, Karel Beneš, Karel Kapsa, Jan Smrz, Pavel Polok, Alexander Hradis, Michal Neverilova, Zuzana Horak, Ales Sabol, Radoslav Stefanik, Michal Jirkovsky, Adam Adamczyk, David Hyner, Petr Hula, Jan Kydlicek, Hynek Computation and Language Artificial Intelligence We present BenCzechMark (BCM), the first comprehensive Czech language benchmark designed for large language models, offering diverse tasks, multiple task formats, and multiple evaluation metrics. Its duel scoring system is grounded in statistical significance theory and uses aggregation across tasks inspired by social preference theory. Our benchmark encompasses 50 challenging tasks, with corresponding test datasets, primarily in native Czech, with 14 newly collected ones. These tasks span 8 categories and cover diverse domains, including historical Czech news, essays from pupils or language learners, and spoken word. Furthermore, we collect and clean BUT-Large Czech Collection, the largest publicly available clean Czech language corpus, and use it for (i) contamination analysis and (ii) continuous pretraining of the first Czech-centric 7B language model with Czech-specific tokenization. We use our model as a baseline for comparison with publicly available multilingual models. Lastly, we release and maintain a leaderboard with existing 50 model submissions, where new model submissions can be made at https://huggingface.co/spaces/CZLC/BenCzechMark. |
| title | BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.17933 |