CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911695187738624 |
|---|---|
| author | Zheng, Congmin Zhu, Jiachen Lin, Jianghao Dai, Xinyi Liu, Weiwen Li, Haoxuan Yu, Yong Zhang, Weinan Yang, Mengyue |
| author_facet | Zheng, Congmin Zhu, Jiachen Lin, Jianghao Dai, Xinyi Liu, Weiwen Li, Haoxuan Yu, Yong Zhang, Weinan Yang, Mengyue |
| contents | Process Reward Models (PRMs) play a central role in evaluating and guiding multi-step reasoning in large language models (LLMs), especially for mathematical problem solving. However, we identify a pervasive length bias in existing PRMs: they tend to assign higher scores to longer reasoning steps, even when the semantic content and logical validity are unchanged. This bias undermines the reliability of reward predictions and leads to overly verbose outputs during inference. To address this issue, we propose CoLD(Counterfactually-Guided Length Debiasing), a unified framework that mitigates length bias through three components: an explicit length-penalty adjustment, a learned bias estimator trained to capture spurious length-related signals, and a joint training strategy that enforces length-invariance in reward predictions. Our approach is grounded in counterfactual reasoning and informed by causal graph analysis. Extensive experiments on MATH500 and GSM-Plus show that CoLD improves accuracy in step selection, and encourages more concise, logically valid reasoning. Furthermore, it consistently improves downstream RL performance and generalizes across domains by mitigating length bias, demonstrating CoLD's strong generalization capability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15698 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning Zheng, Congmin Zhu, Jiachen Lin, Jianghao Dai, Xinyi Liu, Weiwen Li, Haoxuan Yu, Yong Zhang, Weinan Yang, Mengyue Computation and Language Artificial Intelligence Machine Learning Process Reward Models (PRMs) play a central role in evaluating and guiding multi-step reasoning in large language models (LLMs), especially for mathematical problem solving. However, we identify a pervasive length bias in existing PRMs: they tend to assign higher scores to longer reasoning steps, even when the semantic content and logical validity are unchanged. This bias undermines the reliability of reward predictions and leads to overly verbose outputs during inference. To address this issue, we propose CoLD(Counterfactually-Guided Length Debiasing), a unified framework that mitigates length bias through three components: an explicit length-penalty adjustment, a learned bias estimator trained to capture spurious length-related signals, and a joint training strategy that enforces length-invariance in reward predictions. Our approach is grounded in counterfactual reasoning and informed by causal graph analysis. Extensive experiments on MATH500 and GSM-Plus show that CoLD improves accuracy in step selection, and encourages more concise, logically valid reasoning. Furthermore, it consistently improves downstream RL performance and generalizes across domains by mitigating length bias, demonstrating CoLD's strong generalization capability. |
| title | CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2507.15698 |