Value-Guided Search for Efficient Chain-of-Thought Reasoning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908567742709760 |
|---|---|
| author | Wang, Kaiwen Zhou, Jin Peng Chang, Jonathan Gao, Zhaolin Kallus, Nathan Brantley, Kianté Sun, Wen |
| author_facet | Wang, Kaiwen Zhou, Jin Peng Chang, Jonathan Gao, Zhaolin Kallus, Nathan Brantley, Kianté Sun, Wen |
| contents | In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of "step," which is difficult to define for long-context reasoning models. By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (VGS) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-n. Moreover, VGS significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17373 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Value-Guided Search for Efficient Chain-of-Thought Reasoning Wang, Kaiwen Zhou, Jin Peng Chang, Jonathan Gao, Zhaolin Kallus, Nathan Brantley, Kianté Sun, Wen Machine Learning Artificial Intelligence Computation and Language In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of "step," which is difficult to define for long-context reasoning models. By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (VGS) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-n. Moreover, VGS significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced. |
| title | Value-Guided Search for Efficient Chain-of-Thought Reasoning |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2505.17373 |