AfroBench: How Good are Large Language Models on African Languages?
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915332058251264 |
|---|---|
| author | Ojo, Jessica Ogundepo, Odunayo Oladipo, Akintunde Ogueji, Kelechi Lin, Jimmy Stenetorp, Pontus Adelani, David Ifeoluwa |
| author_facet | Ojo, Jessica Ogundepo, Odunayo Oladipo, Akintunde Ogueji, Kelechi Lin, Jimmy Stenetorp, Pontus Adelani, David Ifeoluwa |
| contents | Large-scale multilingual evaluations, such as MEGA, often include only a handful of African languages due to the scarcity of high-quality evaluation data and the limited discoverability of existing African datasets. This lack of representation hinders comprehensive LLM evaluation across a diverse range of languages and tasks. To address these challenges, we introduce AfroBench -- a multi-task benchmark for evaluating the performance of LLMs across 64 African languages, 15 tasks and 22 datasets. AfroBench consists of nine natural language understanding datasets, six text generation datasets, six knowledge and question answering tasks, and one mathematical reasoning task. We present results comparing the performance of prompting LLMs to fine-tuned baselines based on BERT and T5-style models. Our results suggest large gaps in performance between high-resource languages, such as English, and African languages across most tasks; but performance also varies based on the availability of monolingual data resources. Our findings confirm that performance on African languages continues to remain a hurdle for current LLMs, underscoring the need for additional efforts to close this gap.
https://mcgill-nlp.github.io/AfroBench/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_07978 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | AfroBench: How Good are Large Language Models on African Languages? Ojo, Jessica Ogundepo, Odunayo Oladipo, Akintunde Ogueji, Kelechi Lin, Jimmy Stenetorp, Pontus Adelani, David Ifeoluwa Computation and Language Artificial Intelligence Machine Learning Large-scale multilingual evaluations, such as MEGA, often include only a handful of African languages due to the scarcity of high-quality evaluation data and the limited discoverability of existing African datasets. This lack of representation hinders comprehensive LLM evaluation across a diverse range of languages and tasks. To address these challenges, we introduce AfroBench -- a multi-task benchmark for evaluating the performance of LLMs across 64 African languages, 15 tasks and 22 datasets. AfroBench consists of nine natural language understanding datasets, six text generation datasets, six knowledge and question answering tasks, and one mathematical reasoning task. We present results comparing the performance of prompting LLMs to fine-tuned baselines based on BERT and T5-style models. Our results suggest large gaps in performance between high-resource languages, such as English, and African languages across most tasks; but performance also varies based on the availability of monolingual data resources. Our findings confirm that performance on African languages continues to remain a hurdle for current LLMs, underscoring the need for additional efforts to close this gap. https://mcgill-nlp.github.io/AfroBench/ |
| title | AfroBench: How Good are Large Language Models on African Languages? |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2311.07978 |