AlignBench: Benchmarking Chinese Alignment of Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929471989219328 |
|---|---|
| author | Liu, Xiao Lei, Xuanyu Wang, Shengyuan Huang, Yue Feng, Zhuoer Wen, Bosi Cheng, Jiale Ke, Pei Xu, Yifan Tam, Weng Lam Zhang, Xiaohan Sun, Lichao Gu, Xiaotao Wang, Hongning Zhang, Jing Huang, Minlie Dong, Yuxiao Tang, Jie |
| author_facet | Liu, Xiao Lei, Xuanyu Wang, Shengyuan Huang, Yue Feng, Zhuoer Wen, Bosi Cheng, Jiale Ke, Pei Xu, Yifan Tam, Weng Lam Zhang, Xiaohan Sun, Lichao Gu, Xiaotao Wang, Hongning Zhang, Jing Huang, Minlie Dong, Yuxiao Tang, Jie |
| contents | Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, the effective evaluation of alignment for emerging Chinese LLMs is still largely unexplored. To fill in this gap, we introduce AlignBench, a comprehensive multi-dimensional benchmark for evaluating LLMs' alignment in Chinese. We design a human-in-the-loop data curation pipeline, containing eight main categories, 683 real-scenario rooted queries and corresponding human verified references. To ensure the correctness of references, each knowledge-intensive query is accompanied with evidences collected from reliable web sources (including URLs and quotations) by our annotators. For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge~\cite{zheng2023judging} approach with Chain-of-Thought to generate explanations and final ratings, ensuring high reliability and interpretability. All evaluation code, data, and LLM generations are available at \url{https://github.com/THUDM/AlignBench}. Since its release, AlignBench has been adopted by top (Chinese) LLMs for evaluating their alignment capabilities in Chinese, including ChatGLM, Qwen, DeepSeek, Yi, Baichuan, and Abab. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_18743 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | AlignBench: Benchmarking Chinese Alignment of Large Language Models Liu, Xiao Lei, Xuanyu Wang, Shengyuan Huang, Yue Feng, Zhuoer Wen, Bosi Cheng, Jiale Ke, Pei Xu, Yifan Tam, Weng Lam Zhang, Xiaohan Sun, Lichao Gu, Xiaotao Wang, Hongning Zhang, Jing Huang, Minlie Dong, Yuxiao Tang, Jie Computation and Language Artificial Intelligence Machine Learning Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, the effective evaluation of alignment for emerging Chinese LLMs is still largely unexplored. To fill in this gap, we introduce AlignBench, a comprehensive multi-dimensional benchmark for evaluating LLMs' alignment in Chinese. We design a human-in-the-loop data curation pipeline, containing eight main categories, 683 real-scenario rooted queries and corresponding human verified references. To ensure the correctness of references, each knowledge-intensive query is accompanied with evidences collected from reliable web sources (including URLs and quotations) by our annotators. For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge~\cite{zheng2023judging} approach with Chain-of-Thought to generate explanations and final ratings, ensuring high reliability and interpretability. All evaluation code, data, and LLM generations are available at \url{https://github.com/THUDM/AlignBench}. Since its release, AlignBench has been adopted by top (Chinese) LLMs for evaluating their alignment capabilities in Chinese, including ChatGLM, Qwen, DeepSeek, Yi, Baichuan, and Abab. |
| title | AlignBench: Benchmarking Chinese Alignment of Large Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2311.18743 |