Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913084216442880 |
|---|---|
| author | Xue, Chao Wang, Yao Liu, Mengqiao Liang, Di Han, Xingsheng Liu, Peiyang Wu, Xianjie Lu, Chenyao Jiang, Lei Lu, Yu Shi, Haibo Liang, Shuang Peng, Minlong Salim, Flora D. |
| author_facet | Xue, Chao Wang, Yao Liu, Mengqiao Liang, Di Han, Xingsheng Liu, Peiyang Wu, Xianjie Lu, Chenyao Jiang, Lei Lu, Yu Shi, Haibo Liang, Shuang Peng, Minlong Salim, Flora D. |
| contents | Recent advancements in the Generative Reward Model (GRM) have demonstrated its potential to enhance the reasoning abilities of LLMs through Chain-of-Thought (CoT) prompting. Despite these gains, existing implementations of GRM suffer from two critical limitations. First, CoT prompting is applied indiscriminately to all inputs regardless of their inherent complexity. This introduces unnecessary computational costs for tasks amenable to fast, direct inference. Second, existing approaches primarily rely on voting-based mechanisms to evaluate CoT outputs, which often lack granularity and precision in assessing reasoning quality. In this paper, we propose E-GRM, an efficient generative reward modeling framework grounded in model-internal uncertainty. E-GRM leverages the convergence behavior of parallel model generations to estimate uncertainty and selectively trigger CoT reasoning only when needed, without relying on handcrafted features or task-dependent signals. To improve reward fidelity, we introduce a lightweight discriminative scorer trained with a hybrid regression--ranking objective to provide fine-grained evaluation of reasoning paths. Experiments on multiple reasoning benchmarks show that E-GRM substantially reduces inference cost while consistently improving answer accuracy, demonstrating that model-internal uncertainty is an effective and general signal for efficient reasoning-aware reward modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_10072 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty Xue, Chao Wang, Yao Liu, Mengqiao Liang, Di Han, Xingsheng Liu, Peiyang Wu, Xianjie Lu, Chenyao Jiang, Lei Lu, Yu Shi, Haibo Liang, Shuang Peng, Minlong Salim, Flora D. Computation and Language Recent advancements in the Generative Reward Model (GRM) have demonstrated its potential to enhance the reasoning abilities of LLMs through Chain-of-Thought (CoT) prompting. Despite these gains, existing implementations of GRM suffer from two critical limitations. First, CoT prompting is applied indiscriminately to all inputs regardless of their inherent complexity. This introduces unnecessary computational costs for tasks amenable to fast, direct inference. Second, existing approaches primarily rely on voting-based mechanisms to evaluate CoT outputs, which often lack granularity and precision in assessing reasoning quality. In this paper, we propose E-GRM, an efficient generative reward modeling framework grounded in model-internal uncertainty. E-GRM leverages the convergence behavior of parallel model generations to estimate uncertainty and selectively trigger CoT reasoning only when needed, without relying on handcrafted features or task-dependent signals. To improve reward fidelity, we introduce a lightweight discriminative scorer trained with a hybrid regression--ranking objective to provide fine-grained evaluation of reasoning paths. Experiments on multiple reasoning benchmarks show that E-GRM substantially reduces inference cost while consistently improving answer accuracy, demonstrating that model-internal uncertainty is an effective and general signal for efficient reasoning-aware reward modeling. |
| title | Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.10072 |