HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909835221532672 |
|---|---|
| author | Wang, Xinyi Han, Jinyi Jiang, Zishang Li, Tingyun Liang, Jiaqing Jiang, Sihang Dai, Zhaoqian Ma, Shuguang Yu, Fei Xiao, Yanghua |
| author_facet | Wang, Xinyi Han, Jinyi Jiang, Zishang Li, Tingyun Liang, Jiaqing Jiang, Sihang Dai, Zhaoqian Ma, Shuguang Yu, Fei Xiao, Yanghua |
| contents | Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs). However, prevalent methods like GRPO often fail when task difficulty exceeds the model's capacity, leading to reward sparsity and inefficient training. While prior work attempts to mitigate this using off-policy data, such as mixing RL with Supervised Fine-Tuning (SFT) or using hints, they often misguide policy updates In this work, we identify a core issue underlying these failures, which we term low training affinity. This condition arises from a large distributional mismatch between external guidance and the model's policy. To diagnose this, we introduce Affinity, the first quantitative metric for monitoring exploration efficiency and training stability. To improve Affinity, we propose HINT: Helping Ineffective rollouts Navigate Towards effectiveness, an adaptive hinting framework. Instead of providing direct answers, HINT supplies heuristic hints that guide the model to discover solutions on its own, preserving its autonomous reasoning capabilities. Extensive experiments on mathematical reasoning tasks show that HINT consistently outperforms existing methods, achieving state-of-the-art results with models of various scales, while also demonstrating significantly more stable learning and greater data efficiency.Code is available on Github. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09388 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness Wang, Xinyi Han, Jinyi Jiang, Zishang Li, Tingyun Liang, Jiaqing Jiang, Sihang Dai, Zhaoqian Ma, Shuguang Yu, Fei Xiao, Yanghua Machine Learning Computation and Language Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs). However, prevalent methods like GRPO often fail when task difficulty exceeds the model's capacity, leading to reward sparsity and inefficient training. While prior work attempts to mitigate this using off-policy data, such as mixing RL with Supervised Fine-Tuning (SFT) or using hints, they often misguide policy updates In this work, we identify a core issue underlying these failures, which we term low training affinity. This condition arises from a large distributional mismatch between external guidance and the model's policy. To diagnose this, we introduce Affinity, the first quantitative metric for monitoring exploration efficiency and training stability. To improve Affinity, we propose HINT: Helping Ineffective rollouts Navigate Towards effectiveness, an adaptive hinting framework. Instead of providing direct answers, HINT supplies heuristic hints that guide the model to discover solutions on its own, preserving its autonomous reasoning capabilities. Extensive experiments on mathematical reasoning tasks show that HINT consistently outperforms existing methods, achieving state-of-the-art results with models of various scales, while also demonstrating significantly more stable learning and greater data efficiency.Code is available on Github. |
| title | HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2510.09388 |