MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Language
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908639603720192 |
|---|---|
| author | Song, Seyoung Jeong, Seogyeong Kim, Eunsu Jin, Jiho Kim, Dongkwan Shin, Jay Oh, Alice |
| author_facet | Song, Seyoung Jeong, Seogyeong Kim, Eunsu Jin, Jiho Kim, Dongkwan Shin, Jay Oh, Alice |
| contents | Evaluating text generation capabilities of large language models (LLMs) is challenging, particularly for low-resource languages where methods for direct assessment are scarce. We propose MUG-Eval, a novel framework that evaluates LLMs' multilingual generation capabilities by transforming existing benchmarks into conversational tasks and measuring the LLMs' accuracies on those tasks. We specifically designed these conversational tasks to require effective communication in the target language. Then, we simply use task success rate as a proxy for successful conversation generation. Our approach offers two key advantages: it is independent of language-specific NLP tools or annotated datasets, which are limited for most languages, and it does not rely on LLMs-as-judges, whose evaluation quality degrades outside a few high-resource languages. We evaluate 8 LLMs across 30 languages spanning high, mid, and low-resource categories, and we find that MUG-Eval correlates strongly with established benchmarks ($r$ > 0.75) while enabling standardized comparisons across languages and models. Our framework provides a robust and resource-efficient solution for evaluating multilingual generation that can be extended to thousands of languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14395 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Language Song, Seyoung Jeong, Seogyeong Kim, Eunsu Jin, Jiho Kim, Dongkwan Shin, Jay Oh, Alice Computation and Language Artificial Intelligence Evaluating text generation capabilities of large language models (LLMs) is challenging, particularly for low-resource languages where methods for direct assessment are scarce. We propose MUG-Eval, a novel framework that evaluates LLMs' multilingual generation capabilities by transforming existing benchmarks into conversational tasks and measuring the LLMs' accuracies on those tasks. We specifically designed these conversational tasks to require effective communication in the target language. Then, we simply use task success rate as a proxy for successful conversation generation. Our approach offers two key advantages: it is independent of language-specific NLP tools or annotated datasets, which are limited for most languages, and it does not rely on LLMs-as-judges, whose evaluation quality degrades outside a few high-resource languages. We evaluate 8 LLMs across 30 languages spanning high, mid, and low-resource categories, and we find that MUG-Eval correlates strongly with established benchmarks ($r$ > 0.75) while enabling standardized comparisons across languages and models. Our framework provides a robust and resource-efficient solution for evaluating multilingual generation that can be extended to thousands of languages. |
| title | MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Language |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.14395 |