Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916048882630656 |
|---|---|
| author | He, Yuhang Wu, Haodong Liu, Siyi Ge, Hongyu Zhou, Hange Wu, Keyi Zheng, Zhuo Lin, Qihong Zhong, Zixin Zhang, Yongqi |
| author_facet | He, Yuhang Wu, Haodong Liu, Siyi Ge, Hongyu Zhou, Hange Wu, Keyi Zheng, Zhuo Lin, Qihong Zhong, Zixin Zhang, Yongqi |
| contents | Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning ability of Large Language Models (LLMs), but sparse outcome rewards make token-level credit assignment difficult.
We study token-level credit as a reward-conditioned shift from the behavior policy to a hindsight posterior.
In autoregressive RLVR, this shift can be expressed through Conditional Mutual Information (CMI), which shows that token entropy upper-bounds possible hindsight credit.
Entropy, however, indicates capacity rather than update direction, so we introduce the Four Quadrant Decomposition to separate updates by reward polarity and token entropy.
Controlled interventions show that these two factors jointly shape token updates.
Sustained reasoning gains concentrate in signed high-entropy quadrants, whereas low-entropy updates saturate quickly.
Based on this analysis, we propose Hindsight-Aware Policy Optimization (HAPO), a sign-preserving modification to GRPO that performs capacity-guided advantage reallocation.
Experiments on mathematical reasoning benchmarks in two model settings show that HAPO achieves competitive performance among entropy-aware baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_11056 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR He, Yuhang Wu, Haodong Liu, Siyi Ge, Hongyu Zhou, Hange Wu, Keyi Zheng, Zhuo Lin, Qihong Zhong, Zixin Zhang, Yongqi Machine Learning Artificial Intelligence Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning ability of Large Language Models (LLMs), but sparse outcome rewards make token-level credit assignment difficult. We study token-level credit as a reward-conditioned shift from the behavior policy to a hindsight posterior. In autoregressive RLVR, this shift can be expressed through Conditional Mutual Information (CMI), which shows that token entropy upper-bounds possible hindsight credit. Entropy, however, indicates capacity rather than update direction, so we introduce the Four Quadrant Decomposition to separate updates by reward polarity and token entropy. Controlled interventions show that these two factors jointly shape token updates. Sustained reasoning gains concentrate in signed high-entropy quadrants, whereas low-entropy updates saturate quickly. Based on this analysis, we propose Hindsight-Aware Policy Optimization (HAPO), a sign-preserving modification to GRPO that performs capacity-guided advantage reallocation. Experiments on mathematical reasoning benchmarks in two model settings show that HAPO achieves competitive performance among entropy-aware baselines. |
| title | Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.11056 |