Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.20555 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914581576679424 |
|---|---|
| author | Gan, Xingwei Zhu, Ying |
| author_facet | Gan, Xingwei Zhu, Ying |
| contents | We introduce a novel method that averages the logits of a frozen reference policy (e.g., SFT) and a trainable policy, and incorporate the method into Group Relative Policy Optimization (GRPO). In contrast to Reinforcement Learning with Verifiable Rewards (RLVR) methods, our proposal does not involve a Kullback Leibler (KL) regularization or critic; the trainable policy and the reference anchor are coupled through the logit averaging structure to leverage the reasoning expertise of the trainable policy while maintaining the formatting advantage of SFT. Our method is evaluated on MATH, cn-k12, and MMLU, and the results show a higher accuracy or at least comparable accuracy relative to the canonical KL-regularized GRPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_20555 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Complementing reinforcement learning with SFT through logit averaging in the post training of LLMs Gan, Xingwei Zhu, Ying Machine Learning Artificial Intelligence We introduce a novel method that averages the logits of a frozen reference policy (e.g., SFT) and a trainable policy, and incorporate the method into Group Relative Policy Optimization (GRPO). In contrast to Reinforcement Learning with Verifiable Rewards (RLVR) methods, our proposal does not involve a Kullback Leibler (KL) regularization or critic; the trainable policy and the reference anchor are coupled through the logit averaging structure to leverage the reasoning expertise of the trainable policy while maintaining the formatting advantage of SFT. Our method is evaluated on MATH, cn-k12, and MMLU, and the results show a higher accuracy or at least comparable accuracy relative to the canonical KL-regularized GRPO. |
| title | Complementing reinforcement learning with SFT through logit averaging in the post training of LLMs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.20555 |