CFBench: A Comprehensive Constraints-Following Benchmark for LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916720248094720 |
|---|---|
| author | Zhang, Tao Zhu, Chenglin Shen, Yanjun Luo, Wenjing Zhang, Yan Liang, Hao Zhang, Tao Yang, Fan Lin, Mingan Qiao, Yujing Chen, Weipeng Cui, Bin Zhang, Wentao Zhou, Zenan |
| author_facet | Zhang, Tao Zhu, Chenglin Shen, Yanjun Luo, Wenjing Zhang, Yan Liang, Hao Zhang, Tao Yang, Fan Lin, Mingan Qiao, Yujing Chen, Weipeng Cui, Bin Zhang, Wentao Zhou, Zenan |
| contents | The adeptness of Large Language Models (LLMs) in comprehending and following natural language instructions is critical for their deployment in sophisticated real-world applications. Existing evaluations mainly focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user's perspective. To bridge this gap, we propose CFBench, a large-scale Comprehensive Constraints Following Benchmark for LLMs, featuring 1,000 curated samples that cover more than 200 real-life scenarios and over 50 NLP tasks. CFBench meticulously compiles constraints from real-world instructions and constructs an innovative systematic framework for constraint types, which includes 10 primary categories and over 25 subcategories, and ensures each constraint is seamlessly integrated within the instructions. To make certain that the evaluation of LLM outputs aligns with user perceptions, we propose an advanced methodology that integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. Evaluating current leading LLMs on CFBench reveals substantial room for improvement in constraints following, and we further investigate influencing factors and enhancement strategies. The data and code are publicly available at https://github.com/PKU-Baichuan-MLSystemLab/CFBench |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01122 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CFBench: A Comprehensive Constraints-Following Benchmark for LLMs Zhang, Tao Zhu, Chenglin Shen, Yanjun Luo, Wenjing Zhang, Yan Liang, Hao Zhang, Tao Yang, Fan Lin, Mingan Qiao, Yujing Chen, Weipeng Cui, Bin Zhang, Wentao Zhou, Zenan Computation and Language The adeptness of Large Language Models (LLMs) in comprehending and following natural language instructions is critical for their deployment in sophisticated real-world applications. Existing evaluations mainly focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user's perspective. To bridge this gap, we propose CFBench, a large-scale Comprehensive Constraints Following Benchmark for LLMs, featuring 1,000 curated samples that cover more than 200 real-life scenarios and over 50 NLP tasks. CFBench meticulously compiles constraints from real-world instructions and constructs an innovative systematic framework for constraint types, which includes 10 primary categories and over 25 subcategories, and ensures each constraint is seamlessly integrated within the instructions. To make certain that the evaluation of LLM outputs aligns with user perceptions, we propose an advanced methodology that integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. Evaluating current leading LLMs on CFBench reveals substantial room for improvement in constraints following, and we further investigate influencing factors and enhancement strategies. The data and code are publicly available at https://github.com/PKU-Baichuan-MLSystemLab/CFBench |
| title | CFBench: A Comprehensive Constraints-Following Benchmark for LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2408.01122 |