Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.15706 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911391944802304 |
|---|---|
| author | Vij, Akriti Chua, Benjamin Ramiah, Darshini Ng, En Qi Morsidi, Mahran Gangarapu, Naga Nikshith Johnson, Sharmini Wilfred, Vanessa Kumaran, Vikneswaran Lee, Wan Sie Yang, Wenzhuo Zheng, Yongsen Black, Bill Xia, Boming Sun, Frank Zhang, Hao Lu, Qinghua Ma, Suyu Liu, Yue Lo, Chi-kiu Azadi, Fatemeh Nejadgholi, Isar Vajjala, Sowmya Delaborde, Agnes Rolin, Nicolas Seimandi, Tom Murakami, Akiko Ishi, Haruto Sekine, Satoshi Semitsu, Takayuki Sasaki, Tasuku Kinuthia, Angela Wangari, Jean Michie, Michael Kasaon, Stephanie Baek, Hankyul Noh, Jaewon Nam, Kihyuk Seo, Sang Shin, Sungpil Lee, Taewhi Kim, Yongsu Newbold-Harrop, Daisy Wang, Jessica Ghanem, Mahmoud Hong, Vy |
| author_facet | Vij, Akriti Chua, Benjamin Ramiah, Darshini Ng, En Qi Morsidi, Mahran Gangarapu, Naga Nikshith Johnson, Sharmini Wilfred, Vanessa Kumaran, Vikneswaran Lee, Wan Sie Yang, Wenzhuo Zheng, Yongsen Black, Bill Xia, Boming Sun, Frank Zhang, Hao Lu, Qinghua Ma, Suyu Liu, Yue Lo, Chi-kiu Azadi, Fatemeh Nejadgholi, Isar Vajjala, Sowmya Delaborde, Agnes Rolin, Nicolas Seimandi, Tom Murakami, Akiko Ishi, Haruto Sekine, Satoshi Semitsu, Takayuki Sasaki, Tasuku Kinuthia, Angela Wangari, Jean Michie, Michael Kasaon, Stephanie Baek, Hankyul Noh, Jaewon Nam, Kihyuk Seo, Sang Shin, Sungpil Lee, Taewhi Kim, Yongsu Newbold-Harrop, Daisy Wang, Jessica Ghanem, Mahmoud Hong, Vy |
| contents | As frontier AI models are deployed globally, it is essential that their behaviour remains safe and reliable across diverse linguistic and cultural contexts. To examine how current model safeguards hold up in such settings, participants from the International Network for Advanced AI Measurement, Evaluation and Science, including representatives from Singapore, Japan, Australia, Canada, the EU, France, Kenya, South Korea and the UK conducted a joint multilingual evaluation exercise. Led by Singapore AISI, two open-weight models were tested across ten languages spanning high and low resourced groups: Cantonese English, Farsi, French, Japanese, Korean, Kiswahili, Malay, Mandarin Chinese and Telugu. Over 6,000 newly translated prompts were evaluated across five harm categories (privacy, non-violent crime, violent crime, intellectual property and jailbreak robustness), using both LLM-as-a-judge and human annotation.
The exercise shows how safety behaviours can vary across languages. These include differences in safeguard robustness across languages and harm types and variation in evaluator reliability (LLM-as-judge vs. human review). Further, it also generated methodological insights for improving multilingual safety evaluations, such as the need for culturally contextualised translations, stress-tested evaluator prompts and clearer human annotation guidelines. This work represents an initial step toward a shared framework for multilingual safety testing of advanced AI systems and calls for continued collaboration with the wider research community and industry. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_15706 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Improving Methodologies for LLM Evaluations Across Global Languages Vij, Akriti Chua, Benjamin Ramiah, Darshini Ng, En Qi Morsidi, Mahran Gangarapu, Naga Nikshith Johnson, Sharmini Wilfred, Vanessa Kumaran, Vikneswaran Lee, Wan Sie Yang, Wenzhuo Zheng, Yongsen Black, Bill Xia, Boming Sun, Frank Zhang, Hao Lu, Qinghua Ma, Suyu Liu, Yue Lo, Chi-kiu Azadi, Fatemeh Nejadgholi, Isar Vajjala, Sowmya Delaborde, Agnes Rolin, Nicolas Seimandi, Tom Murakami, Akiko Ishi, Haruto Sekine, Satoshi Semitsu, Takayuki Sasaki, Tasuku Kinuthia, Angela Wangari, Jean Michie, Michael Kasaon, Stephanie Baek, Hankyul Noh, Jaewon Nam, Kihyuk Seo, Sang Shin, Sungpil Lee, Taewhi Kim, Yongsu Newbold-Harrop, Daisy Wang, Jessica Ghanem, Mahmoud Hong, Vy Artificial Intelligence As frontier AI models are deployed globally, it is essential that their behaviour remains safe and reliable across diverse linguistic and cultural contexts. To examine how current model safeguards hold up in such settings, participants from the International Network for Advanced AI Measurement, Evaluation and Science, including representatives from Singapore, Japan, Australia, Canada, the EU, France, Kenya, South Korea and the UK conducted a joint multilingual evaluation exercise. Led by Singapore AISI, two open-weight models were tested across ten languages spanning high and low resourced groups: Cantonese English, Farsi, French, Japanese, Korean, Kiswahili, Malay, Mandarin Chinese and Telugu. Over 6,000 newly translated prompts were evaluated across five harm categories (privacy, non-violent crime, violent crime, intellectual property and jailbreak robustness), using both LLM-as-a-judge and human annotation. The exercise shows how safety behaviours can vary across languages. These include differences in safeguard robustness across languages and harm types and variation in evaluator reliability (LLM-as-judge vs. human review). Further, it also generated methodological insights for improving multilingual safety evaluations, such as the need for culturally contextualised translations, stress-tested evaluator prompts and clearer human annotation guidelines. This work represents an initial step toward a shared framework for multilingual safety testing of advanced AI systems and calls for continued collaboration with the wider research community and industry. |
| title | Improving Methodologies for LLM Evaluations Across Global Languages |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.15706 |