Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liu, Kaiyuan, Zhuang, Ziyuan, Bai, Yang, Wang, Bing, Weng, Rongxiang, Ye, Jieping
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization
by: Bai, Yang, et al.
Published: (2026)

Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation
by: Liu, Kaiyuan, et al.
Published: (2025)

Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning
by: Yan, Shaotian, et al.
Published: (2026)

Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller
by: Cai, Min, et al.
Published: (2024)

Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning
by: Zhai, Naixin, et al.
Published: (2026)

Strong Teacher Not Needed? On Distillation in LLM Pretraining
by: Lu, Taiming, et al.
Published: (2026)

Long-Chain Reasoning Distillation via Adaptive Prefix Alignment
by: Liu, Zhenghao, et al.
Published: (2026)

Natural Language Communication with a Teachable Agent
by: Love, Rachel, et al.
Published: (2022)

LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points
by: Zhang, Xuemiao, et al.
Published: (2025)

Are Rationales Necessary and Sufficient? Tuning LLMs for Explainable Misinformation Detection
by: Wang, Bing, et al.
Published: (2026)

Libra: Assessing and Improving Reward Model by Learning to Think
by: Zhou, Meng, et al.
Published: (2025)

Weak-to-Strong Reasoning
by: Yang, Yuqing, et al.
Published: (2024)

Teaching LLM to be Persuasive: Reward-Enhanced Policy Optimization for Alignment from Heterogeneous Rewards
by: Zeng, Xia, et al.
Published: (2025)

Advancing Adversarial Suffix Transfer Learning on Aligned Large Language Models
by: Liu, Hongfu, et al.
Published: (2024)

Length Desensitization in Direct Preference Optimization
by: Liu, Wei, et al.
Published: (2024)

On the Step Length Confounding in LLM Reasoning Data Selection
by: Wang, Bing, et al.
Published: (2026)

Weak-to-Strong Jailbreaking on Large Language Models
by: Zhao, Xuandong, et al.
Published: (2024)

Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model
by: Zhu, Wenhong, et al.
Published: (2024)

PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention
by: Wang, Haonan, et al.
Published: (2025)

Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
by: Wang, Bing, et al.
Published: (2026)

Efficient Reasoning Through Suppression of Self-Affirmation Reflections in Large Reasoning Models
by: Liu, Kaiyuan, et al.
Published: (2025)

Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillation
by: Zhao, Shuai, et al.
Published: (2024)

A Survey on LLM Mid-Training
by: Tu, Chengying, et al.
Published: (2025)

PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization
by: Chen, Mengzhao, et al.
Published: (2024)

Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting
by: Boateng, Emmanuel Aboah, et al.
Published: (2024)

Rumor Detection by Multi-task Suffix Learning based on Time-series Dual Sentiments
by: Liu, Zhiwei, et al.
Published: (2025)

Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization
by: Yang, Wenkai, et al.
Published: (2024)

Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning
by: Sang, Jitao, et al.
Published: (2024)

MoPE: Mixture of Prefix Experts for Zero-Shot Dialogue State Tracking
by: Tang, Tianwen, et al.
Published: (2024)

AmpleGCG-Plus: A Strong Generative Model of Adversarial Suffixes to Jailbreak LLMs with Higher Success Rates in Fewer Attempts
by: Kumar, Vishal, et al.
Published: (2024)

Improving Weak-to-Strong Generalization with Reliability-Aware Alignment
by: Guo, Yue, et al.
Published: (2024)

On-Policy Context Distillation for Language Models
by: Ye, Tianzhu, et al.
Published: (2026)

SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
by: Li, Hao, et al.
Published: (2026)

Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning
by: Ye, Ruimeng, et al.
Published: (2024)

Synthesizing Text-to-SQL Data from Weak and Strong LLMs
by: Yang, Jiaxi, et al.
Published: (2024)

The Alignment Floor: How Persona Customization Breaks Safety in Weakly-Aligned LLMs
by: Zhang, Xing, et al.
Published: (2026)

ROSA-Tuning: Enhancing Long-Context Modeling via Suffix Matching
by: Zheng, Yunao, et al.
Published: (2026)

FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training
by: Xu, Liangyu, et al.
Published: (2025)

Unlocking Implicit Experience: Synthesizing Tool-Use Trajectories from Text
by: Xu, Zhihao, et al.
Published: (2026)

MAIGO: Mitigating Lost-in-Conversation with History-Cleaned On-Policy Self-Distillation
by: Zheng, Haoyu, et al.
Published: (2026)