Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913039669788672 |
|---|---|
| author | Yuan, Youliang Mang, Qiuyang Chen, Jingbang Wan, Hong Liu, Xiaoyuan Xu, Junjielong Huang, Jen-tse Wang, Wenxuan Jiao, Wenxiang He, Pinjia |
| author_facet | Yuan, Youliang Mang, Qiuyang Chen, Jingbang Wan, Hong Liu, Xiaoyuan Xu, Junjielong Huang, Jen-tse Wang, Wenxuan Jiao, Wenxiang He, Pinjia |
| contents | In this paper, we observe that current models are susceptible to reward hacking, leading to a substantial overestimation of a model's reasoning ability. This is evidenced by a high incidence of false positives-solutions that reach the correct answer through an unsound process. Through a systematic analysis with human verification, we establish a taxonomy of these failure modes, identifying patterns like Miracle Steps-abrupt jumps to a correct output without a valid preceding derivation. Probing experiments suggest that these Miracle Steps are linked to answer-recall shortcuts, including memorization from pretraining, where the model accesses the correct answer independently of its reasoning chain. To mitigate this systemic issue, we introduce the Rubric Reward Model (RRM), a process-oriented reward function that evaluates the entire reasoning trajectory against problem-specific rubrics. The RRM explicitly penalizes logical flaws and encourages rigorous deduction. When integrated into an RL pipeline, RRM-based training consistently outperforms outcome-only supervision across four math benchmarks. Notably, it boosts Verified Pass@1024 on AIME2024 from 26.7% to 62.6% and reduces the incidence of Miracle Steps by 71%. Our work demonstrates that rewarding the solution process is crucial for building accurate and reliable models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07774 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards Yuan, Youliang Mang, Qiuyang Chen, Jingbang Wan, Hong Liu, Xiaoyuan Xu, Junjielong Huang, Jen-tse Wang, Wenxuan Jiao, Wenxiang He, Pinjia Computation and Language In this paper, we observe that current models are susceptible to reward hacking, leading to a substantial overestimation of a model's reasoning ability. This is evidenced by a high incidence of false positives-solutions that reach the correct answer through an unsound process. Through a systematic analysis with human verification, we establish a taxonomy of these failure modes, identifying patterns like Miracle Steps-abrupt jumps to a correct output without a valid preceding derivation. Probing experiments suggest that these Miracle Steps are linked to answer-recall shortcuts, including memorization from pretraining, where the model accesses the correct answer independently of its reasoning chain. To mitigate this systemic issue, we introduce the Rubric Reward Model (RRM), a process-oriented reward function that evaluates the entire reasoning trajectory against problem-specific rubrics. The RRM explicitly penalizes logical flaws and encourages rigorous deduction. When integrated into an RL pipeline, RRM-based training consistently outperforms outcome-only supervision across four math benchmarks. Notably, it boosts Verified Pass@1024 on AIME2024 from 26.7% to 62.6% and reduces the incidence of Miracle Steps by 71%. Our work demonstrates that rewarding the solution process is crucial for building accurate and reliable models. |
| title | Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.07774 |