Diagnosing and Mitigating System Bias in Self-Rewarding RL
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912640436011008 |
|---|---|
| author | Tan, Chuyi Yuan, Peiwen Wang, Xinglin Li, Yiwei Feng, Shaoxiong Zhang, Yueqi Shi, Jiayi Zhang, Ji Pan, Boyuan Hu, Yao Li, Kan |
| author_facet | Tan, Chuyi Yuan, Peiwen Wang, Xinglin Li, Yiwei Feng, Shaoxiong Zhang, Yueqi Shi, Jiayi Zhang, Ji Pan, Boyuan Hu, Yao Li, Kan |
| contents | Reinforcement learning with verifiable rewards (RLVR) scales the reasoning ability of large language models (LLMs) but remains bottlenecked by limited labeled samples for continued data scaling. Reinforcement learning with intrinsic rewards (RLIR), where the policy model assigns rewards to its own rollouts, enables sustainable scaling in unlabeled settings, yet its performance and stability lag behind RLVR. We trace this gap to a system bias: the model tends to overestimate its high-confidence rollouts, leading to biased and unstable reward estimation. This bias accumulates as training progresses, with deviations from the oracle drifting toward over-reward, causing unstable training. We characterize this bias using three metrics: $ρ_{\text{noise}}$, $ρ_{\text{selfbias}}$, and $ρ_{\text{symbias}}$. We find that $ρ_{\text{noise}}$ and $ρ_{\text{symbias}}$ impact convergence, while $ρ_{\text{selfbias}}$ amplifies both correct and incorrect updates, leading to instability. To mitigate this, we propose reinforcement learning with ensembled rewards (RLER), which aggregates diverse models and adapts reward interpolation and rollout selection. Extensive experiments show that RLER improves by +13.6% over RLIR and is only 3.6% below RLVR, achieving stable scaling on unlabeled samples, making it highly applicable. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_08977 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Diagnosing and Mitigating System Bias in Self-Rewarding RL Tan, Chuyi Yuan, Peiwen Wang, Xinglin Li, Yiwei Feng, Shaoxiong Zhang, Yueqi Shi, Jiayi Zhang, Ji Pan, Boyuan Hu, Yao Li, Kan Machine Learning Computation and Language Reinforcement learning with verifiable rewards (RLVR) scales the reasoning ability of large language models (LLMs) but remains bottlenecked by limited labeled samples for continued data scaling. Reinforcement learning with intrinsic rewards (RLIR), where the policy model assigns rewards to its own rollouts, enables sustainable scaling in unlabeled settings, yet its performance and stability lag behind RLVR. We trace this gap to a system bias: the model tends to overestimate its high-confidence rollouts, leading to biased and unstable reward estimation. This bias accumulates as training progresses, with deviations from the oracle drifting toward over-reward, causing unstable training. We characterize this bias using three metrics: $ρ_{\text{noise}}$, $ρ_{\text{selfbias}}$, and $ρ_{\text{symbias}}$. We find that $ρ_{\text{noise}}$ and $ρ_{\text{symbias}}$ impact convergence, while $ρ_{\text{selfbias}}$ amplifies both correct and incorrect updates, leading to instability. To mitigate this, we propose reinforcement learning with ensembled rewards (RLER), which aggregates diverse models and adapts reward interpolation and rollout selection. Extensive experiments show that RLER improves by +13.6% over RLIR and is only 3.6% below RLVR, achieving stable scaling on unlabeled samples, making it highly applicable. |
| title | Diagnosing and Mitigating System Bias in Self-Rewarding RL |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2510.08977 |