AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910009790562304 |
|---|---|
| author | Azime, Israel Abebe Angamo, Abenezer Kebede Tamiru, Hana Mekonen Marilign, Dagnachew Mekonnen Slusallek, Philipp Yimam, Seid Muhie Klakow, Dietrich |
| author_facet | Azime, Israel Abebe Angamo, Abenezer Kebede Tamiru, Hana Mekonen Marilign, Dagnachew Mekonnen Slusallek, Philipp Yimam, Seid Muhie Klakow, Dietrich |
| contents | With the growing emphasis on multilingual and cultural evaluation benchmarks for large language models, language and culture are often treated as synonymous, and performance is commonly used as a proxy for a models understanding of a given language. In this work, we argue that such evaluations overlook meaningful cultural variation that exists within a single language. We address this gap by focusing on narratives from different regions of Ethiopia and demonstrate that, despite shared linguistic characteristics, region-specific and domain-specific content substantially influences language evaluation outcomes. To this end, we introduce \textbf{\textit{AmharicStoryQA}}, a long-sequence story question answering benchmark grounded in culturally diverse narratives from Amharic-speaking regions. Using this benchmark, we reveal a significant narrative understanding gap in existing LLMs, highlight pronounced regional differences in evaluation results, and show that supervised fine-tuning yields uneven improvements across regions and evaluation settings. Our findings emphasize the need for culturally grounded benchmarks that go beyond language-level evaluation to more accurately assess and improve narrative understanding in low-resource languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_02774 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic Azime, Israel Abebe Angamo, Abenezer Kebede Tamiru, Hana Mekonen Marilign, Dagnachew Mekonnen Slusallek, Philipp Yimam, Seid Muhie Klakow, Dietrich Computation and Language With the growing emphasis on multilingual and cultural evaluation benchmarks for large language models, language and culture are often treated as synonymous, and performance is commonly used as a proxy for a models understanding of a given language. In this work, we argue that such evaluations overlook meaningful cultural variation that exists within a single language. We address this gap by focusing on narratives from different regions of Ethiopia and demonstrate that, despite shared linguistic characteristics, region-specific and domain-specific content substantially influences language evaluation outcomes. To this end, we introduce \textbf{\textit{AmharicStoryQA}}, a long-sequence story question answering benchmark grounded in culturally diverse narratives from Amharic-speaking regions. Using this benchmark, we reveal a significant narrative understanding gap in existing LLMs, highlight pronounced regional differences in evaluation results, and show that supervised fine-tuning yields uneven improvements across regions and evaluation settings. Our findings emphasize the need for culturally grounded benchmarks that go beyond language-level evaluation to more accurately assess and improve narrative understanding in low-resource languages. |
| title | AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.02774 |