Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.19024 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916709001068544 |
|---|---|
| author | Fan, Jiabin Luo, Guoqing Bowling, Michael Mou, Lili |
| author_facet | Fan, Jiabin Luo, Guoqing Bowling, Michael Mou, Lili |
| contents | We propose a novel k-step return estimation method (called KETCHUP) for Reinforcement Learning(RL)-based knowledge distillation (KD) in text generation tasks. Our idea is to induce a K-step return by using the Bellman Optimality Equation for multiple steps. Theoretical analysis shows that this K-step formulation reduces the variance of the gradient estimates, thus leading to improved RL optimization especially when the student model size is large. Empirical evaluation on three text generation tasks demonstrates that our approach yields superior performance in both standard task metrics and large language model (LLM)-based evaluation. These results suggest that our K-step return induction offers a promising direction for enhancing RL-based KD in LLM research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_19024 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation Fan, Jiabin Luo, Guoqing Bowling, Michael Mou, Lili Computation and Language We propose a novel k-step return estimation method (called KETCHUP) for Reinforcement Learning(RL)-based knowledge distillation (KD) in text generation tasks. Our idea is to induce a K-step return by using the Bellman Optimality Equation for multiple steps. Theoretical analysis shows that this K-step formulation reduces the variance of the gradient estimates, thus leading to improved RL optimization especially when the student model size is large. Empirical evaluation on three text generation tasks demonstrates that our approach yields superior performance in both standard task metrics and large language model (LLM)-based evaluation. These results suggest that our K-step return induction offers a promising direction for enhancing RL-based KD in LLM research. |
| title | KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.19024 |