Cover Image

Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kim, Jeonghye, Luo, Xufang, Kim, Minbeom, Lee, Sangmook, Kim, Dohyung, Jeon, Jiwon, Li, Dongsheng, Yang, Yuqing
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty
by: Kim, Jeonghye, et al.
Published: (2026)

Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR
by: Kim, Jeonghye, et al.
Published: (2026)

ReflAct: World-Grounded Decision Making in LLM Agents via Goal-State Reflection
by: Kim, Jeonghye, et al.
Published: (2025)

Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization
by: Liu, Zeyuan, et al.
Published: (2026)

Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning
by: Lee, Sangmook, et al.
Published: (2025)

Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?
by: Öncel, Fırat, et al.
Published: (2024)

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
by: Jiang, Huiqiang, et al.
Published: (2023)

Self-Refining Language Model Anonymizers via Adversarial Distillation
by: Kim, Kyuyoung, et al.
Published: (2025)

SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks
by: Song, Jiwon, et al.
Published: (2024)

MedRep: Medical Concept Representation for General Electronic Health Record Foundation Models
by: Kim, Junmo, et al.
Published: (2025)

Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?
by: Kang, Deokhyung, et al.
Published: (2025)

Conditional [MASK] Discrete Diffusion Language Model
by: Koh, Hyukhun, et al.
Published: (2024)

Learning to Correct for QA Reasoning with Black-box LLMs
by: Kim, Jaehyung, et al.
Published: (2024)

BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
by: Lee, Gihun, et al.
Published: (2024)

Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
by: Yang, Zhihe, et al.
Published: (2025)

Guiding Frame-Level CTC Alignments Using Self-knowledge Distillation
by: Kim, Eungbeom, et al.
Published: (2024)

CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities
by: Mao, Yujun, et al.
Published: (2024)

LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Framework
by: Kim, Woojun, et al.
Published: (2023)

Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making
by: Kim, Jeonghye, et al.
Published: (2023)

Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning
by: Kim, Jeonghye, et al.
Published: (2024)

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
by: Jiang, Huiqiang, et al.
Published: (2024)

Mitigate Position Bias in Large Language Models via Scaling a Single Dimension
by: Yu, Yijiong, et al.
Published: (2024)

Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study
by: Lim, Junghwan, et al.
Published: (2025)

RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrieval
by: Oh, Minhae, et al.
Published: (2025)

FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration
by: Jo, Dongwon, et al.
Published: (2025)

AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning
by: Kim, Jongsuk, et al.
Published: (2024)

Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs
by: Yang, Jaewoo, et al.
Published: (2024)

AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners
by: Koh, Woosung, et al.
Published: (2025)

L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models
by: Jeon, Hyesung, et al.
Published: (2024)

Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMs
by: Kim, Jaemin, et al.
Published: (2025)

Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
by: Ko, Jongwoo, et al.
Published: (2026)

Reliability-Aware Adaptive Self-Consistency for Efficient Sampling in LLM Reasoning
by: Kim, Junseok, et al.
Published: (2026)

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation
by: Oh, Jungwoo, et al.
Published: (2026)

Why Prompt Optimization Works, and Why It Sometimes Doesn't: A Causal-Inspired Edit-Level Analysis
by: Gong, Shuzhi, et al.
Published: (2026)

SCBench: A KV Cache-Centric Analysis of Long-Context Methods
by: Li, Yucheng, et al.
Published: (2024)

LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
by: Pan, Zhuoshi, et al.
Published: (2024)

Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities
by: Sun, Chung-En, et al.
Published: (2024)

How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs
by: Feng, Guhao, et al.
Published: (2024)

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
by: Zhao, Siyan, et al.
Published: (2026)

Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation
by: Tian, Yijun, et al.
Published: (2024)