Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915578202030080 |
|---|---|
| author | Zeng, Yongcheng Cui, Xinyu Jin, Xuanfa Mi, Qirui Liu, Guoqing Sun, Zexu Yang, Mengyue Li, Dong Ma, Weiyu Yang, Ning Zhao, Jian Hao, Jianye Zhang, Haifeng Wang, Jun |
| author_facet | Zeng, Yongcheng Cui, Xinyu Jin, Xuanfa Mi, Qirui Liu, Guoqing Sun, Zexu Yang, Mengyue Li, Dong Ma, Weiyu Yang, Ning Zhao, Jian Hao, Jianye Zhang, Haifeng Wang, Jun |
| contents | Self-Refinement refers to a model's ability to revise its own responses to produce improved outputs. This capability can also serve as a fundamental mechanism for Self-Improvement, for example, by reconstructing datasets with refined results to enhance intrinsic model performance. However, our comprehensive experiments reveal that large language models (LLMs) show no clear evidence of inherent Self-Refinement and may even experience response quality degradation after Self-Refinement. To address this issue, we propose EVOLVE, a simple and effective framework for eliciting and tracking the evolution of Self-Refinement through iterative training. We first explore optimization methods during training to activate the model's Self-Refinement capability. Then, at inference, we investigate various generation strategies to further enhance and utilize Self-Refinement while supplying the necessary data for training. Through synergistic optimization of training and inference stages, we continually evolve the model's Self-Refinement ability, enabling it to better refine its own responses. Moreover, we demonstrate the potential of leveraging Self-Refinement to achieve broader Self-Improvement of intrinsic model abilities. Experiments show that the evolved Self-Refinement ability enables the Llama-3.1-8B base model to surpass GPT-4o, achieving 62.3% length-controlled and 63.3% raw win rates on AlpacaEval 2, and 50.3% on Arena-Hard. It also generalizes effectively to out-of-domain reasoning tasks, improving performance on mathematical reasoning benchmarks such as GSM8K and MATH. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_05605 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization Zeng, Yongcheng Cui, Xinyu Jin, Xuanfa Mi, Qirui Liu, Guoqing Sun, Zexu Yang, Mengyue Li, Dong Ma, Weiyu Yang, Ning Zhao, Jian Hao, Jianye Zhang, Haifeng Wang, Jun Computation and Language Machine Learning Self-Refinement refers to a model's ability to revise its own responses to produce improved outputs. This capability can also serve as a fundamental mechanism for Self-Improvement, for example, by reconstructing datasets with refined results to enhance intrinsic model performance. However, our comprehensive experiments reveal that large language models (LLMs) show no clear evidence of inherent Self-Refinement and may even experience response quality degradation after Self-Refinement. To address this issue, we propose EVOLVE, a simple and effective framework for eliciting and tracking the evolution of Self-Refinement through iterative training. We first explore optimization methods during training to activate the model's Self-Refinement capability. Then, at inference, we investigate various generation strategies to further enhance and utilize Self-Refinement while supplying the necessary data for training. Through synergistic optimization of training and inference stages, we continually evolve the model's Self-Refinement ability, enabling it to better refine its own responses. Moreover, we demonstrate the potential of leveraging Self-Refinement to achieve broader Self-Improvement of intrinsic model abilities. Experiments show that the evolved Self-Refinement ability enables the Llama-3.1-8B base model to surpass GPT-4o, achieving 62.3% length-controlled and 63.3% raw win rates on AlpacaEval 2, and 50.3% on Arena-Hard. It also generalizes effectively to out-of-domain reasoning tasks, improving performance on mathematical reasoning benchmarks such as GSM8K and MATH. |
| title | Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.05605 |