CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917684126416896 |
|---|---|
| author | LI, Yizhi Zhang, Ge Qu, Xingwei Li, Jiali Li, Zhaoqun Wang, Zekun Li, Hao Yuan, Ruibin Ma, Yinghao Zhang, Kai Zhou, Wangchunshu Liang, Yiming Zhang, Lei Ma, Lei Zhang, Jiajun Li, Zuowen Huang, Stephen W. Lin, Chenghua Fu, Jie |
| author_facet | LI, Yizhi Zhang, Ge Qu, Xingwei Li, Jiali Li, Zhaoqun Wang, Zekun Li, Hao Yuan, Ruibin Ma, Yinghao Zhang, Kai Zhou, Wangchunshu Liang, Yiming Zhang, Lei Ma, Lei Zhang, Jiajun Li, Zuowen Huang, Stephen W. Lin, Chenghua Fu, Jie |
| contents | The advancement of large language models (LLMs) has enhanced the ability to generalize across a wide range of unseen natural language processing (NLP) tasks through instruction-following. Yet, their effectiveness often diminishes in low-resource languages like Chinese, exacerbated by biased evaluations from data leakage, casting doubt on their true generalizability to new linguistic territories. In response, we introduce the Chinese Instruction-Following Benchmark (CIF-Bench), designed to evaluate the zero-shot generalizability of LLMs to the Chinese language. CIF-Bench comprises 150 tasks and 15,000 input-output pairs, developed by native speakers to test complex reasoning and Chinese cultural nuances across 20 categories. To mitigate data contamination, we release only half of the dataset publicly, with the remainder kept private, and introduce diversified instructions to minimize score variance, totaling 45,000 data instances. Our evaluation of 28 selected LLMs reveals a noticeable performance gap, with the best model scoring only 52.9%, highlighting the limitations of LLMs in less familiar language and task contexts. This work not only uncovers the current limitations of LLMs in handling Chinese language tasks but also sets a new standard for future LLM generalizability research, pushing towards the development of more adaptable, culturally informed, and linguistically diverse models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_13109 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models LI, Yizhi Zhang, Ge Qu, Xingwei Li, Jiali Li, Zhaoqun Wang, Zekun Li, Hao Yuan, Ruibin Ma, Yinghao Zhang, Kai Zhou, Wangchunshu Liang, Yiming Zhang, Lei Ma, Lei Zhang, Jiajun Li, Zuowen Huang, Stephen W. Lin, Chenghua Fu, Jie Computation and Language Artificial Intelligence The advancement of large language models (LLMs) has enhanced the ability to generalize across a wide range of unseen natural language processing (NLP) tasks through instruction-following. Yet, their effectiveness often diminishes in low-resource languages like Chinese, exacerbated by biased evaluations from data leakage, casting doubt on their true generalizability to new linguistic territories. In response, we introduce the Chinese Instruction-Following Benchmark (CIF-Bench), designed to evaluate the zero-shot generalizability of LLMs to the Chinese language. CIF-Bench comprises 150 tasks and 15,000 input-output pairs, developed by native speakers to test complex reasoning and Chinese cultural nuances across 20 categories. To mitigate data contamination, we release only half of the dataset publicly, with the remainder kept private, and introduce diversified instructions to minimize score variance, totaling 45,000 data instances. Our evaluation of 28 selected LLMs reveals a noticeable performance gap, with the best model scoring only 52.9%, highlighting the limitations of LLMs in less familiar language and task contexts. This work not only uncovers the current limitations of LLMs in handling Chinese language tasks but also sets a new standard for future LLM generalizability research, pushing towards the development of more adaptable, culturally informed, and linguistically diverse models. |
| title | CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.13109 |