KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916709001068544 |
|---|---|
| author | Fan, Jiabin Luo, Guoqing Bowling, Michael Mou, Lili |
| author_facet | Fan, Jiabin Luo, Guoqing Bowling, Michael Mou, Lili |
| contents | We propose a novel k-step return estimation method (called KETCHUP) for Reinforcement Learning(RL)-based knowledge distillation (KD) in text generation tasks. Our idea is to induce a K-step return by using the Bellman Optimality Equation for multiple steps. Theoretical analysis shows that this K-step formulation reduces the variance of the gradient estimates, thus leading to improved RL optimization especially when the student model size is large. Empirical evaluation on three text generation tasks demonstrates that our approach yields superior performance in both standard task metrics and large language model (LLM)-based evaluation. These results suggest that our K-step return induction offers a promising direction for enhancing RL-based KD in LLM research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_19024 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation Fan, Jiabin Luo, Guoqing Bowling, Michael Mou, Lili Computation and Language We propose a novel k-step return estimation method (called KETCHUP) for Reinforcement Learning(RL)-based knowledge distillation (KD) in text generation tasks. Our idea is to induce a K-step return by using the Bellman Optimality Equation for multiple steps. Theoretical analysis shows that this K-step formulation reduces the variance of the gradient estimates, thus leading to improved RL optimization especially when the student model size is large. Empirical evaluation on three text generation tasks demonstrates that our approach yields superior performance in both standard task metrics and large language model (LLM)-based evaluation. These results suggest that our K-step return induction offers a promising direction for enhancing RL-based KD in LLM research. |
| title | KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.19024 |