Rubric-based On-policy Distillation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910200779243520 |
|---|---|
| author | Fang, Junfeng Hong, Zhepei Zheng, Mao Song, Mingyang Li, Gengsheng Jiang, Houcheng Zhang, Dan Guo, Haiyun Wang, Xiang Chua, Tat-Seng |
| author_facet | Fang, Junfeng Hong, Zhepei Zheng, Mao Song, Mingyang Li, Gengsheng Jiang, Houcheng Zhang, Dan Guo, Haiyun Wang, Xiang Chua, Tat-Seng |
| contents | On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses. To prove it, we introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization. Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10x gain in sample efficiency. These results position rubric-based OPD as a flexible, black-box-compatible alternative to the prevailing logit-based OPD, offering a simple yet strong baseline for scalable distillation across proprietary and open-source LLMs. Code is available at https://github.com/Peregrine123/ROPD_official. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_07396 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Rubric-based On-policy Distillation Fang, Junfeng Hong, Zhepei Zheng, Mao Song, Mingyang Li, Gengsheng Jiang, Houcheng Zhang, Dan Guo, Haiyun Wang, Xiang Chua, Tat-Seng Machine Learning Artificial Intelligence On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses. To prove it, we introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization. Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10x gain in sample efficiency. These results position rubric-based OPD as a flexible, black-box-compatible alternative to the prevailing logit-based OPD, offering a simple yet strong baseline for scalable distillation across proprietary and open-source LLMs. Code is available at https://github.com/Peregrine123/ROPD_official. |
| title | Rubric-based On-policy Distillation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.07396 |