SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916606026711040 |
|---|---|
| author | Liu, Chaoqun Zhang, Wenxuan Ying, Jiahao Aljunied, Mahani Luu, Anh Tuan Bing, Lidong |
| author_facet | Liu, Chaoqun Zhang, Wenxuan Ying, Jiahao Aljunied, Mahani Luu, Anh Tuan Bing, Lidong |
| contents | This study introduces two novel benchmarks, SeaExam and SeaBench, designed to evaluate the capabilities of Large Language Models (LLMs) in Southeast Asian (SEA) application scenarios. Unlike existing multilingual datasets primarily derived from English translations, these benchmarks are constructed based on real-world scenarios from SEA regions. SeaExam draws from regional educational exams to form a comprehensive dataset that encompasses subjects such as local history and literature. In contrast, SeaBench is crafted around multi-turn, open-ended tasks that reflect daily interactions within SEA communities. Our evaluations demonstrate that SeaExam and SeaBench more effectively discern LLM performance on SEA language tasks compared to their translated benchmarks. This highlights the importance of using real-world queries to assess the multilingual capabilities of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06298 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia Liu, Chaoqun Zhang, Wenxuan Ying, Jiahao Aljunied, Mahani Luu, Anh Tuan Bing, Lidong Computation and Language Artificial Intelligence This study introduces two novel benchmarks, SeaExam and SeaBench, designed to evaluate the capabilities of Large Language Models (LLMs) in Southeast Asian (SEA) application scenarios. Unlike existing multilingual datasets primarily derived from English translations, these benchmarks are constructed based on real-world scenarios from SEA regions. SeaExam draws from regional educational exams to form a comprehensive dataset that encompasses subjects such as local history and literature. In contrast, SeaBench is crafted around multi-turn, open-ended tasks that reflect daily interactions within SEA communities. Our evaluations demonstrate that SeaExam and SeaBench more effectively discern LLM performance on SEA language tasks compared to their translated benchmarks. This highlights the importance of using real-world queries to assess the multilingual capabilities of LLMs. |
| title | SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.06298 |