Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.20225 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915948678610944 |
|---|---|
| author | Liu, Yilun Zhao, Chunguang Piao, Mengyao Miao, Lingqi Tao, Shimin He, Minggui Liu, Chenxin Zhang, Li Ma, Hongxia Guo, Jiaxin Liu, Chen Deng, Liqun Wei, Jiansheng Meng, Xiaojun Du, Fanyi Wei, Daimeng Xiao, Yanghua |
| author_facet | Liu, Yilun Zhao, Chunguang Piao, Mengyao Miao, Lingqi Tao, Shimin He, Minggui Liu, Chenxin Zhang, Li Ma, Hongxia Guo, Jiaxin Liu, Chen Deng, Liqun Wei, Jiansheng Meng, Xiaojun Du, Fanyi Wei, Daimeng Xiao, Yanghua |
| contents | Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current benchmarks face three critical limitations: (1) fragmented evaluation dimensions that often neglect deep cultural nuances; (2) insufficient language coverage in subjective tasks relying on low-quality machine translation; and (3) shallow analysis that lacks diagnostic depth beyond simple rankings. To address these, we introduce GaoYao, a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas. First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers (General Multilingual, Cross-cultural, Monocultural) and nine cognitive sub-layers. Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages and synthesizing cross-cultural test sets for 34 cultures, surpassing prior coverage by up to 111%. Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs. Our findings reveal significant geographical performance disparities and distinct gaps between tasks, offering a reliable map for future work. We release the benchmark (https://github.com/lunyiliu/GaoYao). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_20225 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models Liu, Yilun Zhao, Chunguang Piao, Mengyao Miao, Lingqi Tao, Shimin He, Minggui Liu, Chenxin Zhang, Li Ma, Hongxia Guo, Jiaxin Liu, Chen Deng, Liqun Wei, Jiansheng Meng, Xiaojun Du, Fanyi Wei, Daimeng Xiao, Yanghua Computation and Language Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current benchmarks face three critical limitations: (1) fragmented evaluation dimensions that often neglect deep cultural nuances; (2) insufficient language coverage in subjective tasks relying on low-quality machine translation; and (3) shallow analysis that lacks diagnostic depth beyond simple rankings. To address these, we introduce GaoYao, a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas. First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers (General Multilingual, Cross-cultural, Monocultural) and nine cognitive sub-layers. Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages and synthesizing cross-cultural test sets for 34 cultures, surpassing prior coverage by up to 111%. Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs. Our findings reveal significant geographical performance disparities and distinct gaps between tasks, offering a reliable map for future work. We release the benchmark (https://github.com/lunyiliu/GaoYao). |
| title | The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.20225 |