Spurious Rewards: Rethinking Training Signals in RLVR
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918354827083776 |
|---|---|
| author | Shao, Rulin Li, Shuyue Stella Xin, Rui Geng, Scott Wang, Yiping Oh, Sewoong Du, Simon Shaolei Lambert, Nathan Min, Sewon Krishna, Ranjay Tsvetkov, Yulia Hajishirzi, Hannaneh Koh, Pang Wei Zettlemoyer, Luke |
| author_facet | Shao, Rulin Li, Shuyue Stella Xin, Rui Geng, Scott Wang, Yiping Oh, Sewoong Du, Simon Shaolei Lambert, Nathan Min, Sewon Krishna, Ranjay Tsvetkov, Yulia Hajishirzi, Hannaneh Koh, Pang Wei Zettlemoyer, Luke |
| contents | We show that reinforcement learning with verifiable rewards (RLVR) can elicit strong mathematical reasoning in certain language models even with spurious rewards that have little, no, or even negative correlation with the correct answer. For example, RLVR training with GRPO improves MATH-500 performance for Qwen2.5-Math-7B by 21.4 percentage points using randomly assigned rewards, nearly matching the 29.1-point gain from ground-truth rewards. To explain this counterintuitive observation, we show that GRPO exhibits a clipping bias from the clip term, which can amplify high-prior behaviors learned during pretraining even without informative rewards. As a case study, we identify one such behavior in Qwen2.5-Math models, which we call code reasoning -- reasoning in code without actual code execution; code-reasoning frequency increases from 65 percent to over 90 percent with spurious rewards. However, the presence of such amplifiable behaviors is highly model-dependent. In practice, spurious rewards that are effective for Qwen models often fail to produce gains for other model families, such as Llama3 or OLMo2. Our results highlight the importance of validating RL methods across diverse models rather than relying on a single de facto choice: large gains can arise on Qwen models even from random rewards that do not reflect genuine capability improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10947 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Spurious Rewards: Rethinking Training Signals in RLVR Shao, Rulin Li, Shuyue Stella Xin, Rui Geng, Scott Wang, Yiping Oh, Sewoong Du, Simon Shaolei Lambert, Nathan Min, Sewon Krishna, Ranjay Tsvetkov, Yulia Hajishirzi, Hannaneh Koh, Pang Wei Zettlemoyer, Luke Artificial Intelligence Machine Learning We show that reinforcement learning with verifiable rewards (RLVR) can elicit strong mathematical reasoning in certain language models even with spurious rewards that have little, no, or even negative correlation with the correct answer. For example, RLVR training with GRPO improves MATH-500 performance for Qwen2.5-Math-7B by 21.4 percentage points using randomly assigned rewards, nearly matching the 29.1-point gain from ground-truth rewards. To explain this counterintuitive observation, we show that GRPO exhibits a clipping bias from the clip term, which can amplify high-prior behaviors learned during pretraining even without informative rewards. As a case study, we identify one such behavior in Qwen2.5-Math models, which we call code reasoning -- reasoning in code without actual code execution; code-reasoning frequency increases from 65 percent to over 90 percent with spurious rewards. However, the presence of such amplifiable behaviors is highly model-dependent. In practice, spurious rewards that are effective for Qwen models often fail to produce gains for other model families, such as Llama3 or OLMo2. Our results highlight the importance of validating RL methods across diverse models rather than relying on a single de facto choice: large gains can arise on Qwen models even from random rewards that do not reflect genuine capability improvements. |
| title | Spurious Rewards: Rethinking Training Signals in RLVR |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2506.10947 |