FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910473070313472 |
|---|---|
| author | Jiang, Yuxin Wang, Yufei Zeng, Xingshan Zhong, Wanjun Li, Liangyou Mi, Fei Shang, Lifeng Jiang, Xin Liu, Qun Wang, Wei |
| author_facet | Jiang, Yuxin Wang, Yufei Zeng, Xingshan Zhong, Wanjun Li, Liangyou Mi, Fei Shang, Lifeng Jiang, Xin Liu, Qun Wang, Wei |
| contents | The ability to follow instructions is crucial for Large Language Models (LLMs) to handle various real-world applications. Existing benchmarks primarily focus on evaluating pure response quality, rather than assessing whether the response follows constraints stated in the instruction. To fill this research gap, in this paper, we propose FollowBench, a Multi-level Fine-grained Constraints Following Benchmark for LLMs. FollowBench comprehensively includes five different types (i.e., Content, Situation, Style, Format, and Example) of fine-grained constraints. To enable a precise constraint following estimation on diverse difficulties, we introduce a Multi-level mechanism that incrementally adds a single constraint to the initial instruction at each increased level. To assess whether LLMs' outputs have satisfied every individual constraint, we propose to prompt strong LLMs with constraint-evolution paths to handle challenging open-ended instructions. By evaluating 13 closed-source and open-source popular LLMs on FollowBench, we highlight the weaknesses of LLMs in instruction following and point towards potential avenues for future work. The data and code are publicly available at https://github.com/YJiangcm/FollowBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_20410 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models Jiang, Yuxin Wang, Yufei Zeng, Xingshan Zhong, Wanjun Li, Liangyou Mi, Fei Shang, Lifeng Jiang, Xin Liu, Qun Wang, Wei Computation and Language The ability to follow instructions is crucial for Large Language Models (LLMs) to handle various real-world applications. Existing benchmarks primarily focus on evaluating pure response quality, rather than assessing whether the response follows constraints stated in the instruction. To fill this research gap, in this paper, we propose FollowBench, a Multi-level Fine-grained Constraints Following Benchmark for LLMs. FollowBench comprehensively includes five different types (i.e., Content, Situation, Style, Format, and Example) of fine-grained constraints. To enable a precise constraint following estimation on diverse difficulties, we introduce a Multi-level mechanism that incrementally adds a single constraint to the initial instruction at each increased level. To assess whether LLMs' outputs have satisfied every individual constraint, we propose to prompt strong LLMs with constraint-evolution paths to handle challenging open-ended instructions. By evaluating 13 closed-source and open-source popular LLMs on FollowBench, we highlight the weaknesses of LLMs in instruction following and point towards potential avenues for future work. The data and code are publicly available at https://github.com/YJiangcm/FollowBench. |
| title | FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2310.20410 |