Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.04208 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909014877536256 |
|---|---|
| author | Moore, Stephen E. Owusu, Mich-Seth Asare, Akwasi Gyamfi, Lawrence Adu Azunre, Paul Budu, Joel Asiamah, Jonathan Dzobo, Elias Newman, Kelvin Benefo, Edmund O. Datsomor, Gerhardt Appiah, Onesimus Addo Banful, Ama Branoa Kpatah, Lucas Woedem Deishini, Saani Mustapha Ayernor, John |
| author_facet | Moore, Stephen E. Owusu, Mich-Seth Asare, Akwasi Gyamfi, Lawrence Adu Azunre, Paul Budu, Joel Asiamah, Jonathan Dzobo, Elias Newman, Kelvin Benefo, Edmund O. Datsomor, Gerhardt Appiah, Onesimus Addo Banful, Ama Branoa Kpatah, Lucas Woedem Deishini, Saani Mustapha Ayernor, John |
| contents | Large language models (LLMs) have demonstrated impressive multilingual capabilities for well-resourced languages, yet their performance on low-resource African languages remains poorly understood and largely unevaluated. This paper presents Nsanku, a systematic benchmark that evaluates the zero-shot machine translation performance of 19 open-weight and proprietary LLMs across 43 Ghanaian languages paired with English. Evaluation sentences were sourced from the YouVersion Bible platform, providing 300 sentence pairs per language. Two complementary automatic metrics are employed: Bilingual Evaluation Understudy (BLEU) and Character n-gram F-Score (chrF), alongside an average accuracy score and a cross-language consistency dimension. Nsanku represents the most comprehensive LLM translation evaluation for Ghanaian languages conducted to date. Results show that gemini-2.5-flash achieves the highest overall average score of 26.88 (BLEU: 24.60, chrF: 29.16), followed by claude-sonnet-4-5 at 24.87 (BLEU: 22.46, chrF: 27.28) and gpt-4.1 at 23.20 (BLEU: 21.15, chrF: 25.24). Among open-weight models, kimi-k2-instruct-0905 leads at an average score of 20.87. A critical finding from the consistency analysis is that no model and no language reached the Leaders quadrant of high performance and high consistency simultaneously, indicating that current LLMs are not yet reliably usable for Ghanaian language translation at scale. Siwu achieved the highest per-language average score at 25.73 while Nkonya scored lowest at 11.65. Nsanku establishes a publicly available, community-extensible evaluation infrastructure for African language NLP research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_04208 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languages Moore, Stephen E. Owusu, Mich-Seth Asare, Akwasi Gyamfi, Lawrence Adu Azunre, Paul Budu, Joel Asiamah, Jonathan Dzobo, Elias Newman, Kelvin Benefo, Edmund O. Datsomor, Gerhardt Appiah, Onesimus Addo Banful, Ama Branoa Kpatah, Lucas Woedem Deishini, Saani Mustapha Ayernor, John Computation and Language Large language models (LLMs) have demonstrated impressive multilingual capabilities for well-resourced languages, yet their performance on low-resource African languages remains poorly understood and largely unevaluated. This paper presents Nsanku, a systematic benchmark that evaluates the zero-shot machine translation performance of 19 open-weight and proprietary LLMs across 43 Ghanaian languages paired with English. Evaluation sentences were sourced from the YouVersion Bible platform, providing 300 sentence pairs per language. Two complementary automatic metrics are employed: Bilingual Evaluation Understudy (BLEU) and Character n-gram F-Score (chrF), alongside an average accuracy score and a cross-language consistency dimension. Nsanku represents the most comprehensive LLM translation evaluation for Ghanaian languages conducted to date. Results show that gemini-2.5-flash achieves the highest overall average score of 26.88 (BLEU: 24.60, chrF: 29.16), followed by claude-sonnet-4-5 at 24.87 (BLEU: 22.46, chrF: 27.28) and gpt-4.1 at 23.20 (BLEU: 21.15, chrF: 25.24). Among open-weight models, kimi-k2-instruct-0905 leads at an average score of 20.87. A critical finding from the consistency analysis is that no model and no language reached the Leaders quadrant of high performance and high consistency simultaneously, indicating that current LLMs are not yet reliably usable for Ghanaian language translation at scale. Siwu achieved the highest per-language average score at 25.73 while Nkonya scored lowest at 11.65. Nsanku establishes a publicly available, community-extensible evaluation infrastructure for African language NLP research. |
| title | Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.04208 |