Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912450179235840 |
|---|---|
| author | Liao, Baohao Xu, Yuhui Dong, Hanze Li, Junnan Monz, Christof Savarese, Silvio Sahoo, Doyen Xiong, Caiming |
| author_facet | Liao, Baohao Xu, Yuhui Dong, Hanze Li, Junnan Monz, Christof Savarese, Silvio Sahoo, Doyen Xiong, Caiming |
| contents | We introduce Reward-Guided Speculative Decoding (RSD), a novel framework aimed at improving the efficiency of inference in large language models (LLMs). RSD synergistically combines a lightweight draft model with a more powerful target model, incorporating a controlled bias to prioritize high-reward outputs, in contrast to existing speculative decoding methods that enforce strict unbiasedness. RSD employs a process reward model to evaluate intermediate decoding steps and dynamically decide whether to invoke the target model, optimizing the trade-off between computational cost and output quality. We theoretically demonstrate that a threshold-based mixture strategy achieves an optimal balance between resource utilization and performance. Extensive evaluations on challenging reasoning benchmarks, including Olympiad-level tasks, show that RSD delivers significant efficiency gains against decoding with the target model only (up to 4.4x fewer FLOPs), while achieving significant better accuracy than parallel decoding method on average (up to +3.5). These results highlight RSD as a robust and cost-effective approach for deploying LLMs in resource-intensive scenarios. The code is available at https://github.com/BaohaoLiao/RSD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_19324 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reward-Guided Speculative Decoding for Efficient LLM Reasoning Liao, Baohao Xu, Yuhui Dong, Hanze Li, Junnan Monz, Christof Savarese, Silvio Sahoo, Doyen Xiong, Caiming Computation and Language Artificial Intelligence We introduce Reward-Guided Speculative Decoding (RSD), a novel framework aimed at improving the efficiency of inference in large language models (LLMs). RSD synergistically combines a lightweight draft model with a more powerful target model, incorporating a controlled bias to prioritize high-reward outputs, in contrast to existing speculative decoding methods that enforce strict unbiasedness. RSD employs a process reward model to evaluate intermediate decoding steps and dynamically decide whether to invoke the target model, optimizing the trade-off between computational cost and output quality. We theoretically demonstrate that a threshold-based mixture strategy achieves an optimal balance between resource utilization and performance. Extensive evaluations on challenging reasoning benchmarks, including Olympiad-level tasks, show that RSD delivers significant efficiency gains against decoding with the target model only (up to 4.4x fewer FLOPs), while achieving significant better accuracy than parallel decoding method on average (up to +3.5). These results highlight RSD as a robust and cost-effective approach for deploying LLMs in resource-intensive scenarios. The code is available at https://github.com/BaohaoLiao/RSD. |
| title | Reward-Guided Speculative Decoding for Efficient LLM Reasoning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.19324 |