Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.12746 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909456276652032 |
|---|---|
| author | Salido, Eva Sánchez Morante, Roser Gonzalo, Julio Marco, Guillermo Carrillo-de-Albornoz, Jorge Plaza, Laura Amigó, Enrique Fernández, Andrés Benito-Santos, Alejandro Espinosa, Adrián Ghajari Fresno, Victor |
| author_facet | Salido, Eva Sánchez Morante, Roser Gonzalo, Julio Marco, Guillermo Carrillo-de-Albornoz, Jorge Plaza, Laura Amigó, Enrique Fernández, Andrés Benito-Santos, Alejandro Espinosa, Adrián Ghajari Fresno, Victor |
| contents | In this article we present UNED-ACCESS 2024, a bilingual dataset that consists of 1003 multiple-choice questions of university entrance level exams in Spanish and English. Questions are originally formulated in Spanish and translated manually into English, and have not ever been publicly released. A selection of current open-source and proprietary models are evaluated in a uniform zero-shot experimental setting both on the UNED-ACCESS 2024 dataset and on an equivalent subset of MMLU questions. Results show that (i) reasoning questions are challenging for models, (ii) smaller models perform worse than larger models and degrade faster in Spanish than in English and (iii) the performance gap between languages is negligible for the best models and grows up to 37% for smaller models. Model ranking on UNED-ACCESS 2024 is almost identical in English and Spanish, and has also a high correlation (0.98 Pearson) with ranking on MMLU, suggesting that a small dataset is sufficiently diverse and representative to measure performance by discipline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12746 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination Salido, Eva Sánchez Morante, Roser Gonzalo, Julio Marco, Guillermo Carrillo-de-Albornoz, Jorge Plaza, Laura Amigó, Enrique Fernández, Andrés Benito-Santos, Alejandro Espinosa, Adrián Ghajari Fresno, Victor Computation and Language In this article we present UNED-ACCESS 2024, a bilingual dataset that consists of 1003 multiple-choice questions of university entrance level exams in Spanish and English. Questions are originally formulated in Spanish and translated manually into English, and have not ever been publicly released. A selection of current open-source and proprietary models are evaluated in a uniform zero-shot experimental setting both on the UNED-ACCESS 2024 dataset and on an equivalent subset of MMLU questions. Results show that (i) reasoning questions are challenging for models, (ii) smaller models perform worse than larger models and degrade faster in Spanish than in English and (iii) the performance gap between languages is negligible for the best models and grows up to 37% for smaller models. Model ranking on UNED-ACCESS 2024 is almost identical in English and Spanish, and has also a high correlation (0.98 Pearson) with ranking on MMLU, suggesting that a small dataset is sufficiently diverse and representative to measure performance by discipline. |
| title | Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.12746 |