CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917425587421184 |
|---|---|
| author | Lin, Peiqin Lyu, Chenyang Luo, Wenjiang Ye, Haotian Hossain, Md Mehrab Ma, Chunlan Ji, Shaoxiong Samih, Younes Zeng, Bo Jiang, Fan Cao, Yuanbin Duisenbek, Dilda Xun, Adrian Neo Sau Pozdniakova, Daria Misevich, Liubou Marinković, Nevena Nguyen, Ngoc Gia Linh Do, Thi Khanh Linh Sophy, Sarakmatak Hu, Baotian Chen, Guanhua Tang, Gongbo Aji, Alham Fikri Wang, Longyue Luo, Weihua |
| author_facet | Lin, Peiqin Lyu, Chenyang Luo, Wenjiang Ye, Haotian Hossain, Md Mehrab Ma, Chunlan Ji, Shaoxiong Samih, Younes Zeng, Bo Jiang, Fan Cao, Yuanbin Duisenbek, Dilda Xun, Adrian Neo Sau Pozdniakova, Daria Misevich, Liubou Marinković, Nevena Nguyen, Ngoc Gia Linh Do, Thi Khanh Linh Sophy, Sarakmatak Hu, Baotian Chen, Guanhua Tang, Gongbo Aji, Alham Fikri Wang, Longyue Luo, Weihua |
| contents | Large language models (LLMs) are now deployed worldwide, inspiring a surge of benchmarks that measure their multilingual and multicultural abilities. However, these benchmarks prioritize generic language understanding or superficial cultural trivia, leaving the evaluation of grounded tasks -- where models must reason within real-world, context-rich scenarios -- largely unaddressed. To fill this gap, we present CulturALL, a comprehensive and challenging benchmark to assess LLMs' multilingual and multicultural competence on grounded tasks. CulturALL is built via a human--AI collaborative framework: expert annotators ensure appropriate difficulty and factual accuracy, while LLMs lighten the manual workload. By incorporating diverse sources, CulturALL ensures comprehensive scenario coverage. Each item is carefully designed to present a high level of difficulty, making CulturALL challenging. CulturALL contains 2,610 samples in 14 languages from 51 regions, distributed across 16 topics to capture the full breadth of grounded tasks. Experiments show that the best LLM achieves 44.48% accuracy on CulturALL, underscoring substantial room for improvement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_19262 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks Lin, Peiqin Lyu, Chenyang Luo, Wenjiang Ye, Haotian Hossain, Md Mehrab Ma, Chunlan Ji, Shaoxiong Samih, Younes Zeng, Bo Jiang, Fan Cao, Yuanbin Duisenbek, Dilda Xun, Adrian Neo Sau Pozdniakova, Daria Misevich, Liubou Marinković, Nevena Nguyen, Ngoc Gia Linh Do, Thi Khanh Linh Sophy, Sarakmatak Hu, Baotian Chen, Guanhua Tang, Gongbo Aji, Alham Fikri Wang, Longyue Luo, Weihua Computation and Language Artificial Intelligence Large language models (LLMs) are now deployed worldwide, inspiring a surge of benchmarks that measure their multilingual and multicultural abilities. However, these benchmarks prioritize generic language understanding or superficial cultural trivia, leaving the evaluation of grounded tasks -- where models must reason within real-world, context-rich scenarios -- largely unaddressed. To fill this gap, we present CulturALL, a comprehensive and challenging benchmark to assess LLMs' multilingual and multicultural competence on grounded tasks. CulturALL is built via a human--AI collaborative framework: expert annotators ensure appropriate difficulty and factual accuracy, while LLMs lighten the manual workload. By incorporating diverse sources, CulturALL ensures comprehensive scenario coverage. Each item is carefully designed to present a high level of difficulty, making CulturALL challenging. CulturALL contains 2,610 samples in 14 languages from 51 regions, distributed across 16 topics to capture the full breadth of grounded tasks. Experiments show that the best LLM achieves 44.48% accuracy on CulturALL, underscoring substantial room for improvement. |
| title | CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.19262 |