Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912254829527040 |
|---|---|
| author | Zeng, Jie He, Qianyu Ren, Qingyu Liang, Jiaqing Xiao, Yanghua Zhou, Weikang Sun, Zeye Yu, Fei |
| author_facet | Zeng, Jie He, Qianyu Ren, Qingyu Liang, Jiaqing Xiao, Yanghua Zhou, Weikang Sun, Zeye Yu, Fei |
| contents | Real-world instructions with multiple constraints pose a significant challenge to existing large language models (LLMs). An observation is that the LLMs exhibit dramatic performance fluctuation when disturbing the order of the incorporated constraints. Yet, none of the existing works has systematically investigated this position bias problem in the field of multi-constraint instruction following. To bridge this gap, we design a probing task where we quantitatively measure the difficulty distribution of the constraints by a novel Difficulty Distribution Index (CDDI). Through the experimental results, we find that LLMs are more performant when presented with the constraints in a ``hard-to-easy'' order. This preference can be generalized to LLMs with different architecture or different sizes of parameters. Additionally, we conduct an explanation study, providing an intuitive insight into the correlation between the LLM's attention and constraint orders. Our code and dataset are publicly available at https://github.com/meowpass/PBIF. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_17204 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following Zeng, Jie He, Qianyu Ren, Qingyu Liang, Jiaqing Xiao, Yanghua Zhou, Weikang Sun, Zeye Yu, Fei Computation and Language Artificial Intelligence Real-world instructions with multiple constraints pose a significant challenge to existing large language models (LLMs). An observation is that the LLMs exhibit dramatic performance fluctuation when disturbing the order of the incorporated constraints. Yet, none of the existing works has systematically investigated this position bias problem in the field of multi-constraint instruction following. To bridge this gap, we design a probing task where we quantitatively measure the difficulty distribution of the constraints by a novel Difficulty Distribution Index (CDDI). Through the experimental results, we find that LLMs are more performant when presented with the constraints in a ``hard-to-easy'' order. This preference can be generalized to LLMs with different architecture or different sizes of parameters. Additionally, we conduct an explanation study, providing an intuitive insight into the correlation between the LLM's attention and constraint orders. Our code and dataset are publicly available at https://github.com/meowpass/PBIF. |
| title | Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.17204 |