Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Values
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913698728116224 |
|---|---|
| author | Zhang, Hongbo Cui, Han Bao, Guangsheng Yang, Linyi Wang, Jun Zhang, Yue |
| author_facet | Zhang, Hongbo Cui, Han Bao, Guangsheng Yang, Linyi Wang, Jun Zhang, Yue |
| contents | We introduce Direct Value Optimization (DVO), an innovative reinforcement learning framework for enhancing large language models in complex reasoning tasks. Unlike traditional methods relying on preference labels, DVO utilizes value signals at individual reasoning steps, optimizing models via a mean squared error loss. The key benefit of DVO lies in its fine-grained supervision, circumventing the need for labor-intensive human annotations. Target values within the DVO are estimated using either Monte Carlo Tree Search or an outcome value model. Our empirical analysis on both mathematical and commonsense reasoning tasks shows that DVO consistently outperforms existing offline preference optimization techniques, even with fewer training steps. These findings underscore the importance of value signals in advancing reasoning capabilities and highlight DVO as a superior methodology under scenarios lacking explicit human preference information. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13723 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Values Zhang, Hongbo Cui, Han Bao, Guangsheng Yang, Linyi Wang, Jun Zhang, Yue Computation and Language Artificial Intelligence We introduce Direct Value Optimization (DVO), an innovative reinforcement learning framework for enhancing large language models in complex reasoning tasks. Unlike traditional methods relying on preference labels, DVO utilizes value signals at individual reasoning steps, optimizing models via a mean squared error loss. The key benefit of DVO lies in its fine-grained supervision, circumventing the need for labor-intensive human annotations. Target values within the DVO are estimated using either Monte Carlo Tree Search or an outcome value model. Our empirical analysis on both mathematical and commonsense reasoning tasks shows that DVO consistently outperforms existing offline preference optimization techniques, even with fewer training steps. These findings underscore the importance of value signals in advancing reasoning capabilities and highlight DVO as a superior methodology under scenarios lacking explicit human preference information. |
| title | Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Values |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.13723 |