Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.09125 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911400592408576 |
|---|---|
| author | Zhang, Mian Liu, Shujian Dong, Sixun Yin, Ming Hu, Yebowen Wang, Xun Ma, Steven Wang, Song Indurthi, Sathish Reddy Deng, Haoyun Chen, Zhiyu Zoey Song, Kaiqiang |
| author_facet | Zhang, Mian Liu, Shujian Dong, Sixun Yin, Ming Hu, Yebowen Wang, Xun Ma, Steven Wang, Song Indurthi, Sathish Reddy Deng, Haoyun Chen, Zhiyu Zoey Song, Kaiqiang |
| contents | Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors. As tasks grow more challenging, the logic structures embedded in natural language instructions becomes increasingly intricate. However, how well LLMs perform on such logic-rich instructions remains under-explored. We propose LogicIFGen and LogicIFEval. LogicIFGen is a scalable, automated framework for generating verifiable instructions from code functions, which can naturally express rich logic such as conditions, loops, and function calls. We further curate a collection of complex code functions and use LogicIFGen to construct LogicIFEval, a benchmark comprising 426 verifiable logic-rich instructions. Our experiments demonstrate that current state-of-the-art LLMs still struggle to correctly follow the instructions in LogicIFEval. Most LLMs can only follow fewer than 60% of the instructions, revealing significant deficiencies in the instruction-following ability. Code and Benchmark: https://github.com/mianzhang/LogicIF |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09125 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Complex Logical Instruction Generation Zhang, Mian Liu, Shujian Dong, Sixun Yin, Ming Hu, Yebowen Wang, Xun Ma, Steven Wang, Song Indurthi, Sathish Reddy Deng, Haoyun Chen, Zhiyu Zoey Song, Kaiqiang Computation and Language Machine Learning Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors. As tasks grow more challenging, the logic structures embedded in natural language instructions becomes increasingly intricate. However, how well LLMs perform on such logic-rich instructions remains under-explored. We propose LogicIFGen and LogicIFEval. LogicIFGen is a scalable, automated framework for generating verifiable instructions from code functions, which can naturally express rich logic such as conditions, loops, and function calls. We further curate a collection of complex code functions and use LogicIFGen to construct LogicIFEval, a benchmark comprising 426 verifiable logic-rich instructions. Our experiments demonstrate that current state-of-the-art LLMs still struggle to correctly follow the instructions in LogicIFEval. Most LLMs can only follow fewer than 60% of the instructions, revealing significant deficiencies in the instruction-following ability. Code and Benchmark: https://github.com/mianzhang/LogicIF |
| title | Complex Logical Instruction Generation |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2508.09125 |