PACR: Progressively Ascending Confidence Reward for LLM Reasoning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909870024818688 |
|---|---|
| author | Yoon, Eunseop Yoon, Hee Suk Jang, Jaehyun Eom, SooHwan Dai, Qi Luo, Chong Hasegawa-Johnson, Mark A. Yoo, Chang D. |
| author_facet | Yoon, Eunseop Yoon, Hee Suk Jang, Jaehyun Eom, SooHwan Dai, Qi Luo, Chong Hasegawa-Johnson, Mark A. Yoo, Chang D. |
| contents | Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved LLM reasoning, but its sparse, outcome-based reward provides no guidance for intermediate steps, slowing exploration. We propose Progressively Ascending Confidence Reward (PACR), a dense, model-intrinsic reward computed directly from the model's evolving belief in the correct answer. PACR encodes the inductive bias that, along a well-formed reasoning trajectory, the probability of the ground-truth answer should have a generally ascending trend. We provide empirical and theoretical analysis validating that such an inductive bias constrains the exploration search space to regions richer in logically sound reasoning. We demonstrate that PACR accelerates exploration, reaches reward saturation with fewer trajectories, and yields improvements on multiple benchmarks. Our results suggest that dense, model-intrinsic shaping signals can make RLVR training more effective and reliable. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22255 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PACR: Progressively Ascending Confidence Reward for LLM Reasoning Yoon, Eunseop Yoon, Hee Suk Jang, Jaehyun Eom, SooHwan Dai, Qi Luo, Chong Hasegawa-Johnson, Mark A. Yoo, Chang D. Artificial Intelligence Computation and Language Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved LLM reasoning, but its sparse, outcome-based reward provides no guidance for intermediate steps, slowing exploration. We propose Progressively Ascending Confidence Reward (PACR), a dense, model-intrinsic reward computed directly from the model's evolving belief in the correct answer. PACR encodes the inductive bias that, along a well-formed reasoning trajectory, the probability of the ground-truth answer should have a generally ascending trend. We provide empirical and theoretical analysis validating that such an inductive bias constrains the exploration search space to regions richer in logically sound reasoning. We demonstrate that PACR accelerates exploration, reaches reward saturation with fewer trajectories, and yields improvements on multiple benchmarks. Our results suggest that dense, model-intrinsic shaping signals can make RLVR training more effective and reliable. |
| title | PACR: Progressively Ascending Confidence Reward for LLM Reasoning |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2510.22255 |