Evolutionary Guided Decoding: Iterative Value Refinement for LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912628068057088 |
|---|---|
| author | Liu, Zhenhua Li, Lijun Chen, Ruizhe Jiang, Yuxian Zhu, Tong Su, Zhaochen Chen, Wenliang Shao, Jing |
| author_facet | Liu, Zhenhua Li, Lijun Chen, Ruizhe Jiang, Yuxian Zhu, Tong Su, Zhaochen Chen, Wenliang Shao, Jing |
| contents | While guided decoding, especially value-guided methods, has emerged as a cost-effective alternative for controlling language model outputs without re-training models, its effectiveness is limited by the accuracy of the value function. We identify that this inaccuracy stems from a core distributional gap: existing methods train static value functions on trajectories sampled exclusively from the base policy, which inherently confines their training to a narrow and suboptimal view of the potential output space. We propose Iterative Value Refinement, a novel framework designed to bridge this gap. It employs Value Exploration to provide a more comprehensive and robust training signal, complemented by Iterative Self-Refinement, which uses the improved value function from one iteration to guide the generation of higher-quality data for the next. Extensive experiments on text summarization, multi-turn dialogue, and instruction following demonstrate the effectiveness of our framework in aligning language models. Our approach not only achieves alignment but also significantly reduces computational costs by leveraging principled value function optimization for efficient and effective control. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02368 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evolutionary Guided Decoding: Iterative Value Refinement for LLMs Liu, Zhenhua Li, Lijun Chen, Ruizhe Jiang, Yuxian Zhu, Tong Su, Zhaochen Chen, Wenliang Shao, Jing Computation and Language Artificial Intelligence Machine Learning While guided decoding, especially value-guided methods, has emerged as a cost-effective alternative for controlling language model outputs without re-training models, its effectiveness is limited by the accuracy of the value function. We identify that this inaccuracy stems from a core distributional gap: existing methods train static value functions on trajectories sampled exclusively from the base policy, which inherently confines their training to a narrow and suboptimal view of the potential output space. We propose Iterative Value Refinement, a novel framework designed to bridge this gap. It employs Value Exploration to provide a more comprehensive and robust training signal, complemented by Iterative Self-Refinement, which uses the improved value function from one iteration to guide the generation of higher-quality data for the next. Extensive experiments on text summarization, multi-turn dialogue, and instruction following demonstrate the effectiveness of our framework in aligning language models. Our approach not only achieves alignment but also significantly reduces computational costs by leveraging principled value function optimization for efficient and effective control. |
| title | Evolutionary Guided Decoding: Iterative Value Refinement for LLMs |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2503.02368 |