Cover Image

Pre-training Distillation for Large Language Models: A Design Space Exploration

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Peng, Hao, Lv, Xin, Bai, Yushi, Yao, Zijun, Zhang, Jiajie, Hou, Lei, Li, Juanzi
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models
by: Tu, Shangqing, et al.
Published: (2023)

Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models
by: Lin, Nianyi, et al.
Published: (2025)

Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
by: Chen, Jianhui, et al.
Published: (2024)

MMGeoLM: Hard Negative Contrastive Learning for Fine-Grained Geometric Understanding in Large Multimodal Models
by: Sun, Kai, et al.
Published: (2025)

WildReward: Learning Reward Models from In-the-Wild Human Interactions
by: Peng, Hao, et al.
Published: (2026)

LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
by: Bai, Yushi, et al.
Published: (2024)

DeepPrune: Parallel Scaling without Inter-trace Redundancy
by: Tu, Shangqing, et al.
Published: (2025)

Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems
by: Peng, Hao, et al.
Published: (2025)

Constraint Back-translation Improves Complex Instruction Following of Large Language Models
by: Qi, Yunjia, et al.
Published: (2024)

Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders
by: Jing, Yi, et al.
Published: (2026)

AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios
by: Qi, Yunjia, et al.
Published: (2025)

AdaptThink: Reasoning Models Can Learn When to Think
by: Zhang, Jiajie, et al.
Published: (2025)

MRCEval: A Comprehensive, Challenging and Accessible Machine Reading Comprehension Benchmark
by: Ma, Shengkun, et al.
Published: (2025)

ReaRAG: Knowledge-guided Reasoning Enhances Factuality of Large Reasoning Models with Iterative Retrieval Augmented Generation
by: Lee, Zhicheng, et al.
Published: (2025)

A Cause-Effect Look at Alleviating Hallucination of Knowledge-grounded Dialogue Generation
by: Yu, Jifan, et al.
Published: (2024)

LongAlign: A Recipe for Long Context Alignment of Large Language Models
by: Bai, Yushi, et al.
Published: (2024)

Dual-Space Knowledge Distillation for Large Language Models
by: Zhang, Songming, et al.
Published: (2024)

DataMan: Data Manager for Pre-training Large Language Models
by: Peng, Ru, et al.
Published: (2025)

Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models
by: Qian, Chen, et al.
Published: (2024)

Reverse That Number! Decoding Order Matters in Arithmetic Learning
by: Zhang-Li, Daniel, et al.
Published: (2024)

Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models
by: Tang, Lei, et al.
Published: (2025)

StoryAlign: Evaluating and Training Reward Models for Story Generation
by: Xia, Haotian, et al.
Published: (2026)

Are Large Language Models a Good Replacement of Taxonomies?
by: Sun, Yushi, et al.
Published: (2024)

VerIF: Verification Engineering for Reinforcement Learning in Instruction Following
by: Peng, Hao, et al.
Published: (2025)

R-Eval: A Unified Toolkit for Evaluating Domain Knowledge of Retrieval Augmented Large Language Models
by: Tu, Shangqing, et al.
Published: (2024)

Pre-trained Large Language Models for Financial Sentiment Analysis
by: Luo, Wei, et al.
Published: (2024)

Spike No More: Stabilizing the Pre-training of Large Language Models
by: Takase, Sho, et al.
Published: (2023)

How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models
by: Lv, Kangtao, et al.
Published: (2025)

Machine Unlearning of Pre-trained Large Language Models
by: Yao, Jin, et al.
Published: (2024)

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
by: Tang, Shengkun, et al.
Published: (2026)

LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models
by: Tu, Shangqing, et al.
Published: (2025)

DocMamba: Efficient Document Pre-training with State Space Model
by: Hu, Pengfei, et al.
Published: (2024)

Understanding Data Temporality Impact on Large Language Models Pre-training
by: Pilchen, Hippolyte, et al.
Published: (2026)

DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoning
by: Tu, Shangqing, et al.
Published: (2024)

Knowledge-to-Jailbreak: Investigating Knowledge-driven Jailbreaking Attacks for Large Language Models
by: Tu, Shangqing, et al.
Published: (2024)

A Study on Hidden Layer Distillation for Large Language Model Pre-Training
by: Guigon, Maxime, et al.
Published: (2026)

StoryWriter: A Multi-Agent Framework for Long Story Generation
by: Xia, Haotian, et al.
Published: (2025)

LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning
by: Wu, Yuhao, et al.
Published: (2025)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
by: Song, Weixi, et al.
Published: (2023)

Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models
by: Ma, Shengjie, et al.
Published: (2025)