Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.13095 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908874801414144 |
|---|---|
| author | Zhang, Feng Tan, Zezhong Ma, Xinhong Dong, Ziqiang Leng, Xi Zhao, Jianfei Sun, Xin Yang, Yang |
| author_facet | Zhang, Feng Tan, Zezhong Ma, Xinhong Dong, Ziqiang Leng, Xi Zhao, Jianfei Sun, Xin Yang, Yang |
| contents | To address the limited capability expansion and low sample efficiency of Reinforcement Learning (RL), recent methods have integrated ''hints'' into post-training, which are prefix segments of complete reasoning trajectories, aiming for powerful knowledge expansion and reasoning generalization. However, existing hint-based RL methods often neglect the role of difficulty in the hint-ratio schedule and relative-advantage estimation, resulting in unstable learning and excessive imitation of off-policy hints. To address this, we propose ADHint, which explicitly integrates difficulty into both processes to achieve a better trade-off between exploration and imitation. Specifically, we propose Adaptive Hint with Sample Difficulty Prior, which evaluates the difficulty of each sample under the current policy to schedule an appropriate hint ratio for rollout generation. Furthermore, we introduce Consistency-based Gradient Modulation alongside Selective Masking for Hint Preservation, which jointly modulate token-level gradients within hints to prevent biased and destructive updates. Additionally, we propose Advantage Estimation with Rollout Difficulty Posterior, which leverages the relative difficulty of rollouts with and without hints to compute their respective advantages, yielding more balanced updates. Extensive experiments across diverse modalities, model scales, model families, and domains demonstrate that ADHint achieves superior reasoning capabilities and out-of-distribution generalization. Code and datasets will be made publicly available upon paper acceptance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_13095 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning Zhang, Feng Tan, Zezhong Ma, Xinhong Dong, Ziqiang Leng, Xi Zhao, Jianfei Sun, Xin Yang, Yang Computer Vision and Pattern Recognition Machine Learning To address the limited capability expansion and low sample efficiency of Reinforcement Learning (RL), recent methods have integrated ''hints'' into post-training, which are prefix segments of complete reasoning trajectories, aiming for powerful knowledge expansion and reasoning generalization. However, existing hint-based RL methods often neglect the role of difficulty in the hint-ratio schedule and relative-advantage estimation, resulting in unstable learning and excessive imitation of off-policy hints. To address this, we propose ADHint, which explicitly integrates difficulty into both processes to achieve a better trade-off between exploration and imitation. Specifically, we propose Adaptive Hint with Sample Difficulty Prior, which evaluates the difficulty of each sample under the current policy to schedule an appropriate hint ratio for rollout generation. Furthermore, we introduce Consistency-based Gradient Modulation alongside Selective Masking for Hint Preservation, which jointly modulate token-level gradients within hints to prevent biased and destructive updates. Additionally, we propose Advantage Estimation with Rollout Difficulty Posterior, which leverages the relative difficulty of rollouts with and without hints to compute their respective advantages, yielding more balanced updates. Extensive experiments across diverse modalities, model scales, model families, and domains demonstrate that ADHint achieves superior reasoning capabilities and out-of-distribution generalization. Code and datasets will be made publicly available upon paper acceptance. |
| title | ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2512.13095 |