Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.04216 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916883487260672 |
|---|---|
| author | Song, Ruike Song, Zeen Guo, Huijie Qiang, Wenwen |
| author_facet | Song, Ruike Song, Zeen Guo, Huijie Qiang, Wenwen |
| contents | External reasoning systems combine language models with process reward models (PRMs) to select high-quality reasoning paths for complex tasks such as mathematical problem solving. However, these systems are prone to reward hacking, where high-scoring but logically incorrect paths are assigned high scores by the PRMs, leading to incorrect answers. From a causal inference perspective, we attribute this phenomenon primarily to the presence of confounding semantic features. To address it, we propose Causal Reward Adjustment (CRA), a method that mitigates reward hacking by estimating the true reward of a reasoning path. CRA trains sparse autoencoders on the PRM's internal activations to recover interpretable features, then corrects confounding by using backdoor adjustment. Experiments on math solving datasets demonstrate that CRA mitigates reward hacking and improves final accuracy, without modifying the policy model or retraining PRM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_04216 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction Song, Ruike Song, Zeen Guo, Huijie Qiang, Wenwen Machine Learning External reasoning systems combine language models with process reward models (PRMs) to select high-quality reasoning paths for complex tasks such as mathematical problem solving. However, these systems are prone to reward hacking, where high-scoring but logically incorrect paths are assigned high scores by the PRMs, leading to incorrect answers. From a causal inference perspective, we attribute this phenomenon primarily to the presence of confounding semantic features. To address it, we propose Causal Reward Adjustment (CRA), a method that mitigates reward hacking by estimating the true reward of a reasoning path. CRA trains sparse autoencoders on the PRM's internal activations to recover interpretable features, then corrects confounding by using backdoor adjustment. Experiments on math solving datasets demonstrate that CRA mitigates reward hacking and improves final accuracy, without modifying the policy model or retraining PRM. |
| title | Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2508.04216 |