Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.18529 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916023111778304 |
|---|---|
| author | Wei, Zhenlin Jian, Pu Deng, Yingzhuo Wang, Xiaohan Chai, Jiajun Hu, Zhexin Lin, Wei Zhang, Shanbin Yin, Guojun |
| author_facet | Wei, Zhenlin Jian, Pu Deng, Yingzhuo Wang, Xiaohan Chai, Jiajun Hu, Zhexin Lin, Wei Zhang, Shanbin Yin, Guojun |
| contents | The alignment of Large Language Models (LLMs) for complex reasoning heavily relies on Reinforcement Learning with Verifiable Rewards (RLVR). However, standard algorithms like GRPO apply sequence-level rewards uniformly to all tokens, creating a severe credit-assignment bottleneck. While on-policy self-distillation attempts to resolve this by conditioning a self-teacher on privileged contexts, direct exposure to raw oracle solutions often induces over-conditioned teacher distributions, implicit answer leakage, and late-stage training collapse. To overcome these limitations, we propose Asymmetric Meta-Reflective Self-Distillation (AMR-SD). Instead of conditioning directly on raw reference traces, AMR-SD inserts a reflection bottleneck: it compresses diagnostic signals -- from verifier outcomes, peer rollouts, or reference feedback -- into concise, self-generated Socratic hints and critiques. Furthermore, we introduce Causal Information Gain (CIG) with an asymmetric, ReLU-gated threshold to translate these reflections into sparse, highly precise token-level advantage modulations. Combined with temporal annealing, this mechanism preserves the base environmental reward while filtering out distributional noise. Experiments across scientific, mathematical, and tool-use benchmarks demonstrate that AMR-SD significantly outperforms existing baselines, achieving robust long-horizon stability and successfully preventing late-stage collapse. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_18529 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment Wei, Zhenlin Jian, Pu Deng, Yingzhuo Wang, Xiaohan Chai, Jiajun Hu, Zhexin Lin, Wei Zhang, Shanbin Yin, Guojun Artificial Intelligence The alignment of Large Language Models (LLMs) for complex reasoning heavily relies on Reinforcement Learning with Verifiable Rewards (RLVR). However, standard algorithms like GRPO apply sequence-level rewards uniformly to all tokens, creating a severe credit-assignment bottleneck. While on-policy self-distillation attempts to resolve this by conditioning a self-teacher on privileged contexts, direct exposure to raw oracle solutions often induces over-conditioned teacher distributions, implicit answer leakage, and late-stage training collapse. To overcome these limitations, we propose Asymmetric Meta-Reflective Self-Distillation (AMR-SD). Instead of conditioning directly on raw reference traces, AMR-SD inserts a reflection bottleneck: it compresses diagnostic signals -- from verifier outcomes, peer rollouts, or reference feedback -- into concise, self-generated Socratic hints and critiques. Furthermore, we introduce Causal Information Gain (CIG) with an asymmetric, ReLU-gated threshold to translate these reflections into sparse, highly precise token-level advantage modulations. Combined with temporal annealing, this mechanism preserves the base environmental reward while filtering out distributional noise. Experiments across scientific, mathematical, and tool-use benchmarks demonstrate that AMR-SD significantly outperforms existing baselines, achieving robust long-horizon stability and successfully preventing late-stage collapse. |
| title | AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2605.18529 |