SEA-HELM: Southeast Asian Holistic Evaluation of Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912407816765440 |
|---|---|
| author | Susanto, Yosephine Hulagadri, Adithya Venkatadri Montalan, Jann Railey Ngui, Jian Gang Yong, Xian Bin Leong, Weiqi Rengarajan, Hamsawardhini Limkonchotiwat, Peerat Mai, Yifan Tjhi, William Chandra |
| author_facet | Susanto, Yosephine Hulagadri, Adithya Venkatadri Montalan, Jann Railey Ngui, Jian Gang Yong, Xian Bin Leong, Weiqi Rengarajan, Hamsawardhini Limkonchotiwat, Peerat Mai, Yifan Tjhi, William Chandra |
| contents | With the rapid emergence of novel capabilities in Large Language Models (LLMs), the need for rigorous multilingual and multicultural benchmarks that are integrated has become more pronounced. Though existing LLM benchmarks are capable of evaluating specific capabilities of LLMs in English as well as in various mid- to low-resource languages, including those in the Southeast Asian (SEA) region, a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far. Here, we present SEA-HELM, a holistic linguistic and cultural LLM evaluation suite that emphasises SEA languages, comprising five core pillars: (1) NLP Classics, (2) LLM-specifics, (3) SEA Linguistics, (4) SEA Culture, (5) Safety. SEA-HELM currently supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. We also introduce the SEA-HELM leaderboard, which allows users to understand models' multilingual and multicultural performance in a systematic and user-friendly manner. We make the SEA-HELM evaluation code publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14301 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SEA-HELM: Southeast Asian Holistic Evaluation of Language Models Susanto, Yosephine Hulagadri, Adithya Venkatadri Montalan, Jann Railey Ngui, Jian Gang Yong, Xian Bin Leong, Weiqi Rengarajan, Hamsawardhini Limkonchotiwat, Peerat Mai, Yifan Tjhi, William Chandra Computation and Language Artificial Intelligence With the rapid emergence of novel capabilities in Large Language Models (LLMs), the need for rigorous multilingual and multicultural benchmarks that are integrated has become more pronounced. Though existing LLM benchmarks are capable of evaluating specific capabilities of LLMs in English as well as in various mid- to low-resource languages, including those in the Southeast Asian (SEA) region, a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far. Here, we present SEA-HELM, a holistic linguistic and cultural LLM evaluation suite that emphasises SEA languages, comprising five core pillars: (1) NLP Classics, (2) LLM-specifics, (3) SEA Linguistics, (4) SEA Culture, (5) Safety. SEA-HELM currently supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. We also introduce the SEA-HELM leaderboard, which allows users to understand models' multilingual and multicultural performance in a systematic and user-friendly manner. We make the SEA-HELM evaluation code publicly available. |
| title | SEA-HELM: Southeast Asian Holistic Evaluation of Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.14301 |