INJONGO: A Multicultural Intent Detection and Slot-filling Dataset for 16 African Languages
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916613691801600 |
|---|---|
| author | Yu, Hao Alabi, Jesujoba O. Bukula, Andiswa Zhuang, Jian Yun Lee, En-Shiun Annie Guge, Tadesse Kebede Azime, Israel Abebe Buzaaba, Happy Sibanda, Blessing Kudzaishe Kalipe, Godson K. Mukiibi, Jonathan Kabenamualu, Salomon Kabongo Setaka, Mmasibidi Ndolela, Lolwethu Odu, Nkiruka Mabuya, Rooweither Muhammad, Shamsuddeen Hassan Osei, Salomey Samb, Sokhar Murage, Juliet W. Klakow, Dietrich Adelani, David Ifeoluwa |
| author_facet | Yu, Hao Alabi, Jesujoba O. Bukula, Andiswa Zhuang, Jian Yun Lee, En-Shiun Annie Guge, Tadesse Kebede Azime, Israel Abebe Buzaaba, Happy Sibanda, Blessing Kudzaishe Kalipe, Godson K. Mukiibi, Jonathan Kabenamualu, Salomon Kabongo Setaka, Mmasibidi Ndolela, Lolwethu Odu, Nkiruka Mabuya, Rooweither Muhammad, Shamsuddeen Hassan Osei, Salomey Samb, Sokhar Murage, Juliet W. Klakow, Dietrich Adelani, David Ifeoluwa |
| contents | Slot-filling and intent detection are well-established tasks in Conversational AI. However, current large-scale benchmarks for these tasks often exclude evaluations of low-resource languages and rely on translations from English benchmarks, thereby predominantly reflecting Western-centric concepts. In this paper, we introduce Injongo -- a multicultural, open-source benchmark dataset for 16 African languages with utterances generated by native speakers across diverse domains, including banking, travel, home, and dining. Through extensive experiments, we benchmark the fine-tuning multilingual transformer models and the prompting large language models (LLMs), and show the advantage of leveraging African-cultural utterances over Western-centric utterances for improving cross-lingual transfer from the English language. Experimental results reveal that current LLMs struggle with the slot-filling task, with GPT-4o achieving an average performance of 26 F1-score. In contrast, intent detection performance is notably better, with an average accuracy of 70.6%, though it still falls behind the fine-tuning baselines. Compared to the English language, GPT-4o and fine-tuning baselines perform similarly on intent detection, achieving an accuracy of approximately 81%. Our findings suggest that the performance of LLMs is still behind for many low-resource African languages, and more work is needed to further improve their downstream performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_09814 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | INJONGO: A Multicultural Intent Detection and Slot-filling Dataset for 16 African Languages Yu, Hao Alabi, Jesujoba O. Bukula, Andiswa Zhuang, Jian Yun Lee, En-Shiun Annie Guge, Tadesse Kebede Azime, Israel Abebe Buzaaba, Happy Sibanda, Blessing Kudzaishe Kalipe, Godson K. Mukiibi, Jonathan Kabenamualu, Salomon Kabongo Setaka, Mmasibidi Ndolela, Lolwethu Odu, Nkiruka Mabuya, Rooweither Muhammad, Shamsuddeen Hassan Osei, Salomey Samb, Sokhar Murage, Juliet W. Klakow, Dietrich Adelani, David Ifeoluwa Computation and Language Slot-filling and intent detection are well-established tasks in Conversational AI. However, current large-scale benchmarks for these tasks often exclude evaluations of low-resource languages and rely on translations from English benchmarks, thereby predominantly reflecting Western-centric concepts. In this paper, we introduce Injongo -- a multicultural, open-source benchmark dataset for 16 African languages with utterances generated by native speakers across diverse domains, including banking, travel, home, and dining. Through extensive experiments, we benchmark the fine-tuning multilingual transformer models and the prompting large language models (LLMs), and show the advantage of leveraging African-cultural utterances over Western-centric utterances for improving cross-lingual transfer from the English language. Experimental results reveal that current LLMs struggle with the slot-filling task, with GPT-4o achieving an average performance of 26 F1-score. In contrast, intent detection performance is notably better, with an average accuracy of 70.6%, though it still falls behind the fine-tuning baselines. Compared to the English language, GPT-4o and fine-tuning baselines perform similarly on intent detection, achieving an accuracy of approximately 81%. Our findings suggest that the performance of LLMs is still behind for many low-resource African languages, and more work is needed to further improve their downstream performance. |
| title | INJONGO: A Multicultural Intent Detection and Slot-filling Dataset for 16 African Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.09814 |