Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915175461814272 |
|---|---|
| author | Zhang, Yudi Wang, Lu Fang, Meng Du, Yali Huang, Chenghua Wang, Jun Lin, Qingwei Pechenizkiy, Mykola Zhang, Dongmei Rajmohan, Saravan Zhang, Qi |
| author_facet | Zhang, Yudi Wang, Lu Fang, Meng Du, Yali Huang, Chenghua Wang, Jun Lin, Qingwei Pechenizkiy, Mykola Zhang, Dongmei Rajmohan, Saravan Zhang, Qi |
| contents | Distilling large language models (LLMs) typically involves transferring the teacher model's responses through supervised fine-tuning (SFT). However, this approach neglects the potential to distill both data (output content) and reward signals (quality evaluations). Extracting reliable reward signals directly from teacher models is challenging, as LLMs are optimized for generation rather than evaluation, often resulting in biased or inconsistent assessments. To address this limitation, we propose a novel distillation pipeline that transfers both responses and rewards. Our method generates pseudo-rewards through a self-supervised mechanism that leverages the inherent structure of both teacher and student responses, enabling reward learning without explicit external evaluation. The reward model subsequently guides reinforcement learning (RL), allowing iterative refinement of the student model after an SFT warm-up phase. Experiments on GSM8K and MMLU-PRO demonstrate that our method consistently outperforms traditional SFT-based approaches, enabling student models to surpass the performance of their teachers. This work highlights the potential for scalable, efficient distillation through structured self-supervised reward learning, reducing dependence on external reward supervision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19557 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones? Zhang, Yudi Wang, Lu Fang, Meng Du, Yali Huang, Chenghua Wang, Jun Lin, Qingwei Pechenizkiy, Mykola Zhang, Dongmei Rajmohan, Saravan Zhang, Qi Computation and Language Artificial Intelligence Distilling large language models (LLMs) typically involves transferring the teacher model's responses through supervised fine-tuning (SFT). However, this approach neglects the potential to distill both data (output content) and reward signals (quality evaluations). Extracting reliable reward signals directly from teacher models is challenging, as LLMs are optimized for generation rather than evaluation, often resulting in biased or inconsistent assessments. To address this limitation, we propose a novel distillation pipeline that transfers both responses and rewards. Our method generates pseudo-rewards through a self-supervised mechanism that leverages the inherent structure of both teacher and student responses, enabling reward learning without explicit external evaluation. The reward model subsequently guides reinforcement learning (RL), allowing iterative refinement of the student model after an SFT warm-up phase. Experiments on GSM8K and MMLU-PRO demonstrate that our method consistently outperforms traditional SFT-based approaches, enabling student models to surpass the performance of their teachers. This work highlights the potential for scalable, efficient distillation through structured self-supervised reward learning, reducing dependence on external reward supervision. |
| title | Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.19557 |