Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929551168241664 |
|---|---|
| author | Benkirane, Kenza Gongas, Laura Pelles, Shahar Fuchs, Naomi Darmon, Joshua Stenetorp, Pontus Adelani, David Ifeoluwa Sánchez, Eduardo |
| author_facet | Benkirane, Kenza Gongas, Laura Pelles, Shahar Fuchs, Naomi Darmon, Joshua Stenetorp, Pontus Adelani, David Ifeoluwa Sánchez, Eduardo |
| contents | Recent advancements in massively multilingual machine translation systems have significantly enhanced translation accuracy; however, even the best performing systems still generate hallucinations, severely impacting user trust. Detecting hallucinations in Machine Translation (MT) remains a critical challenge, particularly since existing methods excel with High-Resource Languages (HRLs) but exhibit substantial limitations when applied to Low-Resource Languages (LRLs). This paper evaluates sentence-level hallucination detection approaches using Large Language Models (LLMs) and semantic similarity within massively multilingual embeddings. Our study spans 16 language directions, covering HRLs, LRLs, with diverse scripts. We find that the choice of model is essential for performance. On average, for HRLs, Llama3-70B outperforms the previous state of the art by as much as 0.16 MCC (Matthews Correlation Coefficient). However, for LRLs we observe that Claude Sonnet outperforms other LLMs on average by 0.03 MCC. The key takeaway from our study is that LLMs can achieve performance comparable or even better than previously proposed models, despite not being explicitly trained for any machine translation task. However, their advantage is less significant for LRLs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_16470 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models Benkirane, Kenza Gongas, Laura Pelles, Shahar Fuchs, Naomi Darmon, Joshua Stenetorp, Pontus Adelani, David Ifeoluwa Sánchez, Eduardo Computation and Language Artificial Intelligence I.2.7 Recent advancements in massively multilingual machine translation systems have significantly enhanced translation accuracy; however, even the best performing systems still generate hallucinations, severely impacting user trust. Detecting hallucinations in Machine Translation (MT) remains a critical challenge, particularly since existing methods excel with High-Resource Languages (HRLs) but exhibit substantial limitations when applied to Low-Resource Languages (LRLs). This paper evaluates sentence-level hallucination detection approaches using Large Language Models (LLMs) and semantic similarity within massively multilingual embeddings. Our study spans 16 language directions, covering HRLs, LRLs, with diverse scripts. We find that the choice of model is essential for performance. On average, for HRLs, Llama3-70B outperforms the previous state of the art by as much as 0.16 MCC (Matthews Correlation Coefficient). However, for LRLs we observe that Claude Sonnet outperforms other LLMs on average by 0.03 MCC. The key takeaway from our study is that LLMs can achieve performance comparable or even better than previously proposed models, despite not being explicitly trained for any machine translation task. However, their advantage is less significant for LRLs. |
| title | Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models |
| topic | Computation and Language Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2407.16470 |