Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2509.04292 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911138469380096 |
|---|---|
| author | Zhang, Qinyan Lei, Xinping Miao, Ruijie Fu, Yu Fan, Haojie Chang, Le Hou, Jiafan Zhang, Dingling Hou, Zhongfei Yang, Ziqiang Pu, Changxin Hu, Fei Liu, Jingkai Liu, Mengyun Liu, Yang Gao, Xiang Liu, Jiaheng Yang, Tong Wang, Zaiyuan Zhang, Ge Huang, Wenhao |
| author_facet | Zhang, Qinyan Lei, Xinping Miao, Ruijie Fu, Yu Fan, Haojie Chang, Le Hou, Jiafan Zhang, Dingling Hou, Zhongfei Yang, Ziqiang Pu, Changxin Hu, Fei Liu, Jingkai Liu, Mengyun Liu, Yang Gao, Xiang Liu, Jiaheng Yang, Tong Wang, Zaiyuan Zhang, Ge Huang, Wenhao |
| contents | Large Language Models (LLMs) achieve strong performance on diverse tasks but often exhibit cognitive inertia, struggling to follow instructions that conflict with the standardized patterns learned during supervised fine-tuning (SFT). To evaluate this limitation, we propose Inverse IFEval, a benchmark that measures models Counter-intuitive Abilitytheir capacity to override training-induced biases and comply with adversarial instructions. Inverse IFEval introduces eight types of such challenges, including Question Correction, Intentional Textual Flaws, Code without Comments, and Counterfactual Answering. Using a human-in-the-loop pipeline, we construct a dataset of 1012 high-quality Chinese and English questions across 23 domains, evaluated under an optimized LLM-as-a-Judge framework. Experiments on existing leading LLMs demonstrate the necessity of our proposed Inverse IFEval benchmark. Our findings emphasize that future alignment efforts should not only pursue fluency and factual correctness but also account for adaptability under unconventional contexts. We hope that Inverse IFEval serves as both a diagnostic tool and a foundation for developing methods that mitigate cognitive inertia, reduce overfitting to narrow patterns, and ultimately enhance the instruction-following reliability of LLMs in diverse and unpredictable real-world scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_04292 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions? Zhang, Qinyan Lei, Xinping Miao, Ruijie Fu, Yu Fan, Haojie Chang, Le Hou, Jiafan Zhang, Dingling Hou, Zhongfei Yang, Ziqiang Pu, Changxin Hu, Fei Liu, Jingkai Liu, Mengyun Liu, Yang Gao, Xiang Liu, Jiaheng Yang, Tong Wang, Zaiyuan Zhang, Ge Huang, Wenhao Computation and Language Large Language Models (LLMs) achieve strong performance on diverse tasks but often exhibit cognitive inertia, struggling to follow instructions that conflict with the standardized patterns learned during supervised fine-tuning (SFT). To evaluate this limitation, we propose Inverse IFEval, a benchmark that measures models Counter-intuitive Abilitytheir capacity to override training-induced biases and comply with adversarial instructions. Inverse IFEval introduces eight types of such challenges, including Question Correction, Intentional Textual Flaws, Code without Comments, and Counterfactual Answering. Using a human-in-the-loop pipeline, we construct a dataset of 1012 high-quality Chinese and English questions across 23 domains, evaluated under an optimized LLM-as-a-Judge framework. Experiments on existing leading LLMs demonstrate the necessity of our proposed Inverse IFEval benchmark. Our findings emphasize that future alignment efforts should not only pursue fluency and factual correctness but also account for adaptability under unconventional contexts. We hope that Inverse IFEval serves as both a diagnostic tool and a foundation for developing methods that mitigate cognitive inertia, reduce overfitting to narrow patterns, and ultimately enhance the instruction-following reliability of LLMs in diverse and unpredictable real-world scenarios. |
| title | Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.04292 |