Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.07591 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911593305997312 |
|---|---|
| author | Ye, Junjie Huang, Caishuang Chen, Zhuohan Fu, Wenjie Yang, Chenyuan Yang, Leyi Wu, Yilong Wang, Peng Zhou, Meng Yang, Xiaolong Gui, Tao Zhang, Qi Shi, Zhongchao Fan, Jianping Huang, Xuanjing |
| author_facet | Ye, Junjie Huang, Caishuang Chen, Zhuohan Fu, Wenjie Yang, Chenyuan Yang, Leyi Wu, Yilong Wang, Peng Zhou, Meng Yang, Xiaolong Gui, Tao Zhang, Qi Shi, Zhongchao Fan, Jianping Huang, Xuanjing |
| contents | Instruction following refers to the ability of large language models (LLMs) to generate outputs that satisfy all specified constraints. Existing research has primarily focused on constraint categories, offering limited evaluation dimensions and little guidance for improving instruction-following abilities. To address this gap, we introduce MulDimIF, a multi-dimensional constraint framework encompassing three constraint patterns, four constraint categories, and four difficulty levels. Based on this framework, we design a controllable instruction generation pipeline. Through constraint expansion, conflict detection, and instruction rewriting, we construct 9,106 code-verifiable samples. We evaluate 18 LLMs from six model families and find marked performance differences across constraint settings. For instance, average accuracy decreases from 80.82% at Level I to 36.76% at Level IV. Moreover, training with data generated by our framework significantly improves instruction following without compromising general performance. In-depth analysis indicates that these gains stem largely from parameter updates in attention modules, which strengthen constraint recognition and adherence. Code and data are available in https://github.com/Junjie-Ye/MulDimIF. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_07591 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models Ye, Junjie Huang, Caishuang Chen, Zhuohan Fu, Wenjie Yang, Chenyuan Yang, Leyi Wu, Yilong Wang, Peng Zhou, Meng Yang, Xiaolong Gui, Tao Zhang, Qi Shi, Zhongchao Fan, Jianping Huang, Xuanjing Computation and Language Artificial Intelligence Instruction following refers to the ability of large language models (LLMs) to generate outputs that satisfy all specified constraints. Existing research has primarily focused on constraint categories, offering limited evaluation dimensions and little guidance for improving instruction-following abilities. To address this gap, we introduce MulDimIF, a multi-dimensional constraint framework encompassing three constraint patterns, four constraint categories, and four difficulty levels. Based on this framework, we design a controllable instruction generation pipeline. Through constraint expansion, conflict detection, and instruction rewriting, we construct 9,106 code-verifiable samples. We evaluate 18 LLMs from six model families and find marked performance differences across constraint settings. For instance, average accuracy decreases from 80.82% at Level I to 36.76% at Level IV. Moreover, training with data generated by our framework significantly improves instruction following without compromising general performance. In-depth analysis indicates that these gains stem largely from parameter updates in attention modules, which strengthen constraint recognition and adherence. Code and data are available in https://github.com/Junjie-Ye/MulDimIF. |
| title | MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.07591 |