Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.03717 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915712837091328 |
|---|---|
| author | Cui, Jin Guo, Jiaqi Zhou, Jiepeng Yang, Ruixuan Lu, Jiayi Xu, Jiajun Song, Jiangcheng Zhao, Boran Ren, Pengju |
| author_facet | Cui, Jin Guo, Jiaqi Zhou, Jiepeng Yang, Ruixuan Lu, Jiayi Xu, Jiajun Song, Jiangcheng Zhao, Boran Ren, Pengju |
| contents | While Large Language Models (LLMs) have emerged with remarkable capabilities in complex tasks through Chain-of-Thought reasoning, practical resource constraints have sparked interest in transferring these abilities to smaller models. However, achieving both domain performance and cross-domain generalization remains challenging. Existing approaches typically restrict students to following a single golden rationale and treat different reasoning paths independently. Due to distinct inductive biases and intrinsic preferences, alongside the student's evolving capacity and reasoning preferences during training, a teacher's "optimal" rationale could act as out-of-distribution noise. This misalignment leads to a degeneration of the student's latent reasoning distribution, causing suboptimal performance. To bridge this gap, we propose MIND, a capability-adaptive framework that transitions distillation from passive mimicry to active cognitive construction. We synthesize diverse teacher perspectives through a novel "Teaching Assistant" network. By employing a Feedback-Driven Inertia Calibration mechanism, this network utilizes inertia-filtered training loss to align supervision with the student's current adaptability, effectively enhancing performance while mitigating catastrophic forgetting. Extensive experiments demonstrate that MIND achieves state-of-the-art performance on both in-distribution and out-of-distribution benchmarks, and our sophisticated latent space analysis further confirms the mechanism of reasoning ability internalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03717 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation Cui, Jin Guo, Jiaqi Zhou, Jiepeng Yang, Ruixuan Lu, Jiayi Xu, Jiajun Song, Jiangcheng Zhao, Boran Ren, Pengju Computation and Language While Large Language Models (LLMs) have emerged with remarkable capabilities in complex tasks through Chain-of-Thought reasoning, practical resource constraints have sparked interest in transferring these abilities to smaller models. However, achieving both domain performance and cross-domain generalization remains challenging. Existing approaches typically restrict students to following a single golden rationale and treat different reasoning paths independently. Due to distinct inductive biases and intrinsic preferences, alongside the student's evolving capacity and reasoning preferences during training, a teacher's "optimal" rationale could act as out-of-distribution noise. This misalignment leads to a degeneration of the student's latent reasoning distribution, causing suboptimal performance. To bridge this gap, we propose MIND, a capability-adaptive framework that transitions distillation from passive mimicry to active cognitive construction. We synthesize diverse teacher perspectives through a novel "Teaching Assistant" network. By employing a Feedback-Driven Inertia Calibration mechanism, this network utilizes inertia-filtered training loss to align supervision with the student's current adaptability, effectively enhancing performance while mitigating catastrophic forgetting. Extensive experiments demonstrate that MIND achieves state-of-the-art performance on both in-distribution and out-of-distribution benchmarks, and our sophisticated latent space analysis further confirms the mechanism of reasoning ability internalization. |
| title | MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.03717 |