Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.12911 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913800902410240 |
|---|---|
| author | Shi, Weijie Ju, Chengyi Liu, Chengzhong Ji, Jiaming Zhang, Jipeng Zhang, Ruiyuan Zhu, Jia Xu, Jiajie Yang, Yaodong Han, Sirui Guo, Yike |
| author_facet | Shi, Weijie Ju, Chengyi Liu, Chengzhong Ji, Jiaming Zhang, Jipeng Zhang, Ruiyuan Zhu, Jia Xu, Jiajie Yang, Yaodong Han, Sirui Guo, Yike |
| contents | Do Large Language Models (LLMs) hold positions that conflict with your country's values? Occasionally they do! However, existing works primarily focus on ethical reviews, failing to capture the diversity of national values, which encompass broader policy, legal, and moral considerations. Furthermore, current benchmarks that rely on spectrum tests using manually designed questionnaires are not easily scalable.
To address these limitations, we introduce NaVAB, a comprehensive benchmark to evaluate the alignment of LLMs with the values of five major nations: China, the United States, the United Kingdom, France, and Germany. NaVAB implements a national value extraction pipeline to efficiently construct value assessment datasets. Specifically, we propose a modeling procedure with instruction tagging to process raw data sources, a screening process to filter value-related topics and a generation process with a Conflict Reduction mechanism to filter non-conflicting values.We conduct extensive experiments on various LLMs across countries, and the results provide insights into assisting in the identification of misaligned scenarios. Moreover, we demonstrate that NaVAB can be combined with alignment techniques to effectively reduce value concerns by aligning LLMs' values with the target country. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_12911 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Benchmarking Multi-National Value Alignment for Large Language Models Shi, Weijie Ju, Chengyi Liu, Chengzhong Ji, Jiaming Zhang, Jipeng Zhang, Ruiyuan Zhu, Jia Xu, Jiajie Yang, Yaodong Han, Sirui Guo, Yike Computation and Language Artificial Intelligence Do Large Language Models (LLMs) hold positions that conflict with your country's values? Occasionally they do! However, existing works primarily focus on ethical reviews, failing to capture the diversity of national values, which encompass broader policy, legal, and moral considerations. Furthermore, current benchmarks that rely on spectrum tests using manually designed questionnaires are not easily scalable. To address these limitations, we introduce NaVAB, a comprehensive benchmark to evaluate the alignment of LLMs with the values of five major nations: China, the United States, the United Kingdom, France, and Germany. NaVAB implements a national value extraction pipeline to efficiently construct value assessment datasets. Specifically, we propose a modeling procedure with instruction tagging to process raw data sources, a screening process to filter value-related topics and a generation process with a Conflict Reduction mechanism to filter non-conflicting values.We conduct extensive experiments on various LLMs across countries, and the results provide insights into assisting in the identification of misaligned scenarios. Moreover, we demonstrate that NaVAB can be combined with alignment techniques to effectively reduce value concerns by aligning LLMs' values with the target country. |
| title | Benchmarking Multi-National Value Alignment for Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.12911 |