Copertina

Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Xu, Wenda, Han, Rujun, Wang, Zifeng, Le, Long T., Madeka, Dhruv, Li, Lei, Wang, William Yang, Agarwal, Rishabh, Lee, Chen-Yu, Pfister, Tomas
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Dual-Forward Path Teacher Knowledge Distillation: Bridging the Capacity Gap Between Teacher and Student
di: Li, Tong, et al.
Pubblicazione: (2025)

DistillSpec: Improving Speculative Decoding via Knowledge Distillation
di: Zhou, Yongchao, et al.
Pubblicazione: (2023)

Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting
di: Wang, Zilong, et al.
Pubblicazione: (2024)

Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation
di: Sun, Zengkui, et al.
Pubblicazione: (2025)

Towards Compute-Optimal Many-Shot In-Context Learning
di: Golchin, Shahriar, et al.
Pubblicazione: (2025)

Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation
di: Ouyang, Siru, et al.
Pubblicazione: (2024)

Reverse Thinking Makes LLMs Stronger Reasoners
di: Chen, Justin Chih-Yao, et al.
Pubblicazione: (2024)

Don't Throw Away Data: Better Sequence Knowledge Distillation
di: Wang, Jun, et al.
Pubblicazione: (2024)

BridgeTA: Bridging the Representation Gap in Knowledge Distillation via Teacher Assistant for Bird's Eye View Map Segmentation
di: Kim, Beomjun, et al.
Pubblicazione: (2025)

Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
di: Deng, Yihe, et al.
Pubblicazione: (2025)

Student-Oriented Teacher Knowledge Refinement for Knowledge Distillation
di: Shen, Chaomin, et al.
Pubblicazione: (2024)

CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation
di: Zhang, Zherui, et al.
Pubblicazione: (2025)

RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
di: Li, Gaotang, et al.
Pubblicazione: (2026)

ASPEST: Bridging the Gap Between Active Learning and Selective Prediction
di: Chen, Jiefeng, et al.
Pubblicazione: (2023)

Do Students Debias Like Teachers? On the Distillability of Bias Mitigation Methods
di: Cheng, Jiali, et al.
Pubblicazione: (2025)

MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillation
di: Li, Hui, et al.
Pubblicazione: (2025)

SAGE: Steerable Agentic Data Generation for Deep Search with Execution Feedback
di: Xu, Fangyuan, et al.
Pubblicazione: (2026)

CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation
di: Hsu, I-Hung, et al.
Pubblicazione: (2024)

Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias
di: Zhang, Jianhua, et al.
Pubblicazione: (2024)

Efficient Interleaved Speech Modeling through Knowledge Distillation
di: Nouriborji, Mohammadmahdi, et al.
Pubblicazione: (2025)

Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model
di: Wang, Yu, et al.
Pubblicazione: (2026)

Bridging the Gap in Missing Modalities: Leveraging Knowledge Distillation and Style Matching for Brain Tumor Segmentation
di: Zhu, Shenghao, et al.
Pubblicazione: (2025)

BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphs
di: Wang, Zifeng, et al.
Pubblicazione: (2023)

INSIGHT: Bridging the Student-Teacher Gap in Times of Large Language Models
di: Thys, Jarne, et al.
Pubblicazione: (2025)

Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillation
di: Gui, Lujun, et al.
Pubblicazione: (2024)

Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation
di: Yin, Fan, et al.
Pubblicazione: (2025)

From Teacher to Student: Tracking Memorization Through Model Distillation
di: Singh, Simardeep
Pubblicazione: (2025)

Toward Student-Oriented Teacher Network Training For Knowledge Distillation
di: Dong, Chengyu, et al.
Pubblicazione: (2022)

Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
di: Yang, Zhaorui, et al.
Pubblicazione: (2024)

CodecLM: Aligning Language Models with Tailored Synthetic Data
di: Wang, Zifeng, et al.
Pubblicazione: (2024)

SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging
di: Pourreza, Mohammadreza, et al.
Pubblicazione: (2024)

Bridging Through Absence: How Comeback Researchers Bridge Knowledge Gaps Through Structural Re-emergence
di: Chakraborty, Somyajit, et al.
Pubblicazione: (2026)

Bridging the Gap: Unpacking the Hidden Challenges in Knowledge Distillation for Online Ranking Systems
di: Khani, Nikhil, et al.
Pubblicazione: (2024)

Bridging the Domain Gap in Equation Distillation with Reinforcement Feedback
di: Ying, Wangyang, et al.
Pubblicazione: (2025)

Scaling Laws for Imitation Learning in Single-Agent Games
di: Tuyls, Jens, et al.
Pubblicazione: (2023)

Neural Coordination and Capacity Control for Inventory Management
di: Eisenach, Carson, et al.
Pubblicazione: (2024)

In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
di: Tan, Zhen, et al.
Pubblicazione: (2025)

Generalizing Teacher Networks for Effective Knowledge Distillation Across Student Architectures
di: Binici, Kuluhan, et al.
Pubblicazione: (2024)

LMBot: Distilling Graph Knowledge into Language Model for Graph-less Deployment in Twitter Bot Detection
di: Cai, Zijian, et al.
Pubblicazione: (2023)

Difficulty-guided Sampling: Bridging the Target Gap between Dataset Distillation and Downstream Tasks
di: Li, Mingzhuo, et al.
Pubblicazione: (2026)