Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.26601 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911475075907584 |
|---|---|
| author | Whitehouse, Chenxi Ruder, Sebastian Lin, Tony Kurylo, Oksana Takagi, Haruka Lam, Janice Busetto, Nicolò Diaz, Denise Guzmán, Francisco |
| author_facet | Whitehouse, Chenxi Ruder, Sebastian Lin, Tony Kurylo, Oksana Takagi, Haruka Lam, Janice Busetto, Nicolò Diaz, Denise Guzmán, Francisco |
| contents | Ensuring native-like quality of large language model (LLM) responses across many languages is challenging. To address this, we introduce MENLO, a framework that operationalizes the evaluation of native-like response quality based on audience design-inspired mechanisms. Using MENLO, we create a dataset of 6,423 human-annotated prompt-response preference pairs covering four quality dimensions with high inter-annotator agreement in 47 language varieties. Our evaluation reveals that zero-shot LLM judges benefit significantly from pairwise evaluation and our structured annotation rubrics, yet they still underperform human annotators on our dataset. We demonstrate substantial improvements through fine-tuning with reinforcement learning, reward shaping, and multi-task learning approaches. Additionally, we show that RL-trained judges can serve as generative reward models to enhance LLMs' multilingual proficiency, though discrepancies with human judgment remain. Our findings suggest promising directions for scalable multilingual evaluation and preference alignment. We release our dataset and evaluation framework to support further research in multilingual LLM evaluation (https://huggingface.co/datasets/facebook/menlo). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_26601 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages Whitehouse, Chenxi Ruder, Sebastian Lin, Tony Kurylo, Oksana Takagi, Haruka Lam, Janice Busetto, Nicolò Diaz, Denise Guzmán, Francisco Computation and Language Artificial Intelligence Machine Learning Ensuring native-like quality of large language model (LLM) responses across many languages is challenging. To address this, we introduce MENLO, a framework that operationalizes the evaluation of native-like response quality based on audience design-inspired mechanisms. Using MENLO, we create a dataset of 6,423 human-annotated prompt-response preference pairs covering four quality dimensions with high inter-annotator agreement in 47 language varieties. Our evaluation reveals that zero-shot LLM judges benefit significantly from pairwise evaluation and our structured annotation rubrics, yet they still underperform human annotators on our dataset. We demonstrate substantial improvements through fine-tuning with reinforcement learning, reward shaping, and multi-task learning approaches. Additionally, we show that RL-trained judges can serve as generative reward models to enhance LLMs' multilingual proficiency, though discrepancies with human judgment remain. Our findings suggest promising directions for scalable multilingual evaluation and preference alignment. We release our dataset and evaluation framework to support further research in multilingual LLM evaluation (https://huggingface.co/datasets/facebook/menlo). |
| title | MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2509.26601 |