Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.20752 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911079734444032 |
|---|---|
| author | Alfano, Carlo Marjani, Aymen Al Jonke, Zeno Mantrach, Amin Mansour, Saab Federico, Marcello |
| author_facet | Alfano, Carlo Marjani, Aymen Al Jonke, Zeno Mantrach, Amin Mansour, Saab Federico, Marcello |
| contents | The growing use of large language models (LLMs) has increased the need for automatic evaluation systems, particularly to address the challenge of information hallucination. Although existing faithfulness evaluation approaches have shown promise, they are predominantly English-focused and often require expensive human-labeled training data for fine-tuning specialized models. As LLMs see increased adoption in multilingual contexts, there is a need for accurate faithfulness evaluators that can operate across languages without extensive labeled data. This paper presents Self-Taught Evaluators for Multilingual Faithfulness, a framework that learns exclusively from synthetic multilingual summarization data while leveraging cross-lingual transfer learning. Through experiments comparing language-specific and mixed-language fine-tuning approaches, we demonstrate a consistent relationship between an LLM's general language capabilities and its performance in language-specific evaluation tasks. Our framework shows improvements over existing baselines, including state-of-the-art English evaluators and machine translation-based approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_20752 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multilingual Self-Taught Faithfulness Evaluators Alfano, Carlo Marjani, Aymen Al Jonke, Zeno Mantrach, Amin Mansour, Saab Federico, Marcello Computation and Language Machine Learning The growing use of large language models (LLMs) has increased the need for automatic evaluation systems, particularly to address the challenge of information hallucination. Although existing faithfulness evaluation approaches have shown promise, they are predominantly English-focused and often require expensive human-labeled training data for fine-tuning specialized models. As LLMs see increased adoption in multilingual contexts, there is a need for accurate faithfulness evaluators that can operate across languages without extensive labeled data. This paper presents Self-Taught Evaluators for Multilingual Faithfulness, a framework that learns exclusively from synthetic multilingual summarization data while leveraging cross-lingual transfer learning. Through experiments comparing language-specific and mixed-language fine-tuning approaches, we demonstrate a consistent relationship between an LLM's general language capabilities and its performance in language-specific evaluation tasks. Our framework shows improvements over existing baselines, including state-of-the-art English evaluators and machine translation-based approaches. |
| title | Multilingual Self-Taught Faithfulness Evaluators |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2507.20752 |