Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915425366835200 |
|---|---|
| author | Ren, Qingyu He, Qianyu Zhang, Bowei Zeng, Jie Liang, Jiaqing Xiao, Yanghua Zhou, Weikang Sun, Zeye Yu, Fei |
| author_facet | Ren, Qingyu He, Qianyu Zhang, Bowei Zeng, Jie Liang, Jiaqing Xiao, Yanghua Zhou, Weikang Sun, Zeye Yu, Fei |
| contents | Reasoning models excel in complex problem solving but exhibit a concerning trade off between reasoning capabilities and instruction following abilities. Existing approaches for improving instruction following rely on stronger external models, creating methodological bottlenecks and practical limitations including increased costs and accessibility constraints. We propose a self-supervised RL framework that leverages reasoning models' own internal signals to improve instruction following capabilities without external supervision. Extensive experiments demonstrate that our framework significantly improves instruction following capabilities while maintaining reasoning performance, offering a scalable and cost-effective approach to enhance instruction following in reasoning models. The data and code are publicly available at https://github.com/Rainier-rq/verl-if. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_02150 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following Ren, Qingyu He, Qianyu Zhang, Bowei Zeng, Jie Liang, Jiaqing Xiao, Yanghua Zhou, Weikang Sun, Zeye Yu, Fei Artificial Intelligence Reasoning models excel in complex problem solving but exhibit a concerning trade off between reasoning capabilities and instruction following abilities. Existing approaches for improving instruction following rely on stronger external models, creating methodological bottlenecks and practical limitations including increased costs and accessibility constraints. We propose a self-supervised RL framework that leverages reasoning models' own internal signals to improve instruction following capabilities without external supervision. Extensive experiments demonstrate that our framework significantly improves instruction following capabilities while maintaining reasoning performance, offering a scalable and cost-effective approach to enhance instruction following in reasoning models. The data and code are publicly available at https://github.com/Rainier-rq/verl-if. |
| title | Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2508.02150 |