CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916361447407616 |
|---|---|
| author | Yu, Linhao Leng, Yongqi Huang, Yufei Wu, Shang Liu, Haixin Ji, Xinmeng Zhao, Jiahui Song, Jinwang Cui, Tingting Cheng, Xiaoqing Liu, Tao Xiong, Deyi |
| author_facet | Yu, Linhao Leng, Yongqi Huang, Yufei Wu, Shang Liu, Haixin Ji, Xinmeng Zhao, Jiahui Song, Jinwang Cui, Tingting Cheng, Xiaoqing Liu, Tao Xiong, Deyi |
| contents | What a large language model (LLM) would respond in ethically relevant context? In this paper, we curate a large benchmark CMoralEval for morality evaluation of Chinese LLMs. The data sources of CMoralEval are two-fold: 1) a Chinese TV program discussing Chinese moral norms with stories from the society and 2) a collection of Chinese moral anomies from various newspapers and academic papers on morality. With these sources, we aim to create a moral evaluation dataset characterized by diversity and authenticity. We develop a morality taxonomy and a set of fundamental moral principles that are not only rooted in traditional Chinese culture but also consistent with contemporary societal norms. To facilitate efficient construction and annotation of instances in CMoralEval, we establish a platform with AI-assisted instance generation to streamline the annotation process. These help us curate CMoralEval that encompasses both explicit moral scenarios (14,964 instances) and moral dilemma scenarios (15,424 instances), each with instances from different data sources. We conduct extensive experiments with CMoralEval to examine a variety of Chinese LLMs. Experiment results demonstrate that CMoralEval is a challenging benchmark for Chinese LLMs. The dataset is publicly available at \url{https://github.com/tjunlp-lab/CMoralEval}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_09819 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Models Yu, Linhao Leng, Yongqi Huang, Yufei Wu, Shang Liu, Haixin Ji, Xinmeng Zhao, Jiahui Song, Jinwang Cui, Tingting Cheng, Xiaoqing Liu, Tao Xiong, Deyi Computation and Language Artificial Intelligence What a large language model (LLM) would respond in ethically relevant context? In this paper, we curate a large benchmark CMoralEval for morality evaluation of Chinese LLMs. The data sources of CMoralEval are two-fold: 1) a Chinese TV program discussing Chinese moral norms with stories from the society and 2) a collection of Chinese moral anomies from various newspapers and academic papers on morality. With these sources, we aim to create a moral evaluation dataset characterized by diversity and authenticity. We develop a morality taxonomy and a set of fundamental moral principles that are not only rooted in traditional Chinese culture but also consistent with contemporary societal norms. To facilitate efficient construction and annotation of instances in CMoralEval, we establish a platform with AI-assisted instance generation to streamline the annotation process. These help us curate CMoralEval that encompasses both explicit moral scenarios (14,964 instances) and moral dilemma scenarios (15,424 instances), each with instances from different data sources. We conduct extensive experiments with CMoralEval to examine a variety of Chinese LLMs. Experiment results demonstrate that CMoralEval is a challenging benchmark for Chinese LLMs. The dataset is publicly available at \url{https://github.com/tjunlp-lab/CMoralEval}. |
| title | CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.09819 |