Precision over Diversity: High-Precision Reward Generalizes to Robust Instruction Following
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917198109343744 |
|---|---|
| author | Zeng, Yirong Liu, Yufei Ding, Xiao Hou, Yutai Wang, Yuxian Song, Haonan Ning, Wu Tu, Dandan Zhang, Qixun Cai, Bibo He, Yuxiang Liu, Ting |
| author_facet | Zeng, Yirong Liu, Yufei Ding, Xiao Hou, Yutai Wang, Yuxian Song, Haonan Ning, Wu Tu, Dandan Zhang, Qixun Cai, Bibo He, Yuxiang Liu, Ting |
| contents | A central belief in scaling reinforcement learning with verifiable rewards for instruction following (IF) tasks is that, a diverse mixture of verifiable hard and unverifiable soft constraints is essential for generalizing to unseen instructions. In this work, we challenge this prevailing consensus through a systematic empirical investigation. Counter-intuitively, we find that models trained on hard-only constraints consistently outperform those trained on mixed datasets. Extensive experiments reveal that reward precision, rather than constraint diversity, is the primary driver of effective alignment. The LLM judge suffers from a low recall rate in detecting false response, which leads to severe reward hacking, thereby undermining the benefits of diversity. Furthermore, analysis of the attention mechanism reveals that high-precision rewards develop a transferable meta-skill for IF. Motivated by these insights, we propose a simple yet effective data-centric refinement strategy that prioritizes reward precision. Evaluated on five benchmarks, our approach outperforms competitive baselines by 13.4\% in performance while achieving a 58\% reduction in training time, maintaining strong generalization beyond instruction following. Our findings advocate for a paradigm shift: moving away from the indiscriminate pursuit of data diversity toward high-precision rewards. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_04954 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Precision over Diversity: High-Precision Reward Generalizes to Robust Instruction Following Zeng, Yirong Liu, Yufei Ding, Xiao Hou, Yutai Wang, Yuxian Song, Haonan Ning, Wu Tu, Dandan Zhang, Qixun Cai, Bibo He, Yuxiang Liu, Ting Machine Learning Artificial Intelligence A central belief in scaling reinforcement learning with verifiable rewards for instruction following (IF) tasks is that, a diverse mixture of verifiable hard and unverifiable soft constraints is essential for generalizing to unseen instructions. In this work, we challenge this prevailing consensus through a systematic empirical investigation. Counter-intuitively, we find that models trained on hard-only constraints consistently outperform those trained on mixed datasets. Extensive experiments reveal that reward precision, rather than constraint diversity, is the primary driver of effective alignment. The LLM judge suffers from a low recall rate in detecting false response, which leads to severe reward hacking, thereby undermining the benefits of diversity. Furthermore, analysis of the attention mechanism reveals that high-precision rewards develop a transferable meta-skill for IF. Motivated by these insights, we propose a simple yet effective data-centric refinement strategy that prioritizes reward precision. Evaluated on five benchmarks, our approach outperforms competitive baselines by 13.4\% in performance while achieving a 58\% reduction in training time, maintaining strong generalization beyond instruction following. Our findings advocate for a paradigm shift: moving away from the indiscriminate pursuit of data diversity toward high-precision rewards. |
| title | Precision over Diversity: High-Precision Reward Generalizes to Robust Instruction Following |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.04954 |