Cover Image

Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Fu, Yao, Yu, Yin, Han, Xiaotian, Li, Runchao, Long, Xianxuan, Yu, Haotian, Li, Pan
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMs
by: Fu, Yao, et al.
Published: (2025)

FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression
by: Li, Runchao, et al.
Published: (2025)

When Truthful Representations Flip Under Deceptive Instructions?
by: Long, Xianxuan, et al.
Published: (2025)

Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs
by: Fu, Yao, et al.
Published: (2025)

Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation
by: Wang, Yibo, et al.
Published: (2025)

Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures
by: Nguyen, Dang, et al.
Published: (2024)

Privately Learning from Graphs with Applications in Fine-tuning Large Language Models
by: Yin, Haoteng, et al.
Published: (2024)

Demystifying Hybrid Thinking: Can LLMs Truly Switch Between Think and No-Think?
by: Wang, Shouren, et al.
Published: (2025)

Exploring Memorization in Fine-tuned Language Models
by: Zeng, Shenglai, et al.
Published: (2023)

Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration
by: Fu, Wenjie, et al.
Published: (2023)

Selecting Large Language Model to Fine-tune via Rectified Scaling Law
by: Lin, Haowei, et al.
Published: (2024)

MiniLLM: On-Policy Distillation of Large Language Models
by: Gu, Yuxian, et al.
Published: (2023)

PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning
by: Zhu, Xuekai, et al.
Published: (2023)

Fine-tuning Large Language Models with Sequential Instructions
by: Hu, Hanxu, et al.
Published: (2024)

Sensitivity of Small Language Models to Fine-tuning Data Contamination
by: Scaria, Nicy, et al.
Published: (2025)

Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
by: Jiang, Yuxuan, et al.
Published: (2026)

Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
by: Yang, Zhaorui, et al.
Published: (2024)

Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning
by: Tan, Zhaoxuan, et al.
Published: (2024)

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
by: Chen, Yukang, et al.
Published: (2023)

SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
by: Yang, Yu, et al.
Published: (2024)

Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Models
by: Chen, Pin-Yu, et al.
Published: (2025)

Sparse Matrix in Large Language Model Fine-tuning
by: He, Haoze, et al.
Published: (2024)

Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models
by: Wang, Yu, et al.
Published: (2025)

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
by: Hu, Shengding, et al.
Published: (2024)

Mini-Giants: "Small" Language Models and Open Source Win-Win
by: Zhou, Zhengping, et al.
Published: (2023)

Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models
by: Kim, Kyeonghyun, et al.
Published: (2025)

Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning
by: Juneja, Gurusha, et al.
Published: (2023)

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
by: Zhang, Renrui, et al.
Published: (2023)

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources
by: Bai, Jiamu, et al.
Published: (2024)

Assertion Detection Large Language Model In-context Learning LoRA Fine-tuning
by: Ji, Yuelyu, et al.
Published: (2024)

MiniPLM: Knowledge Distillation for Pre-Training Language Models
by: Gu, Yuxian, et al.
Published: (2024)

Path-Lock Expert: Separating Reasoning Mode in Hybrid Thinking via Architecture-Level Separation
by: Wang, Shouren, et al.
Published: (2026)

Efficient End-to-end Language Model Fine-tuning on Graphs
by: Xue, Rui, et al.
Published: (2023)

Fine-tuning Language Models with Generative Adversarial Reward Modelling
by: Yu, Zhang Ze, et al.
Published: (2023)

Demystifying Instruction Mixing for Fine-tuning Large Language Models
by: Wang, Renxi, et al.
Published: (2023)

Fine-tuning Small Language Models as Efficient Enterprise Search Relevance Labelers
by: Kang, Yue, et al.
Published: (2026)

When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models
by: Zheng, Yingming, et al.
Published: (2025)

GraphGhost: Tracing Structures Behind Large Language Models
by: Dai, Xinnan, et al.
Published: (2025)

Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
by: Zhou, Yiyang, et al.
Published: (2024)

Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets
by: Li, Dongyue, et al.
Published: (2025)