Image de couverture de livre

Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Wu, Yu-Hang, Liu, Qin-Yuan, Zhao, Qiu-Yang, Jiang, Bo, Yang, Jiang-Feng, Cong, Qing-Wei
Format:	Preprint
Publié:	2026
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs
par: Hong, Yun, et autres
Publié: (2026)

Analysing The Impact of Sequence Composition on Language Model Pre-Training
par: Zhao, Yu, et autres
Publié: (2024)

Continual Pre-Training is (not) What You Need in Domain Adaption
par: Chen, Pin-Er, et autres
Publié: (2025)

LoopRPT: Reinforcement Pre-Training for Looped Language Models
par: Tang, Guo, et autres
Publié: (2026)

Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale
par: Zheng, Wenzhen, et autres
Publié: (2024)

Reinforcement Learning on Pre-Training Data
par: Li, Siheng, et autres
Publié: (2025)

Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training
par: Zhuang, Yuchen, et autres
Publié: (2025)

PowLU: An Activation Function for Stable Pre-Training of LLMs
par: Jiang, Peijie, et autres
Publié: (2026)

Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantization
par: Hajimolahoseini, Habib, et autres
Publié: (2023)

Evolution of Concepts in Language Model Pre-Training
par: Ge, Xuyang, et autres
Publié: (2025)

Reinforcement Pre-Training
par: Dong, Qingxiu, et autres
Publié: (2025)

Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping
par: Chen, Yao, et autres
Publié: (2026)

Autoregressive Pre-Training on Pixels and Texts
par: Chai, Yekun, et autres
Publié: (2024)

Revise, Don't Freeze: Sampler-Matched Training for Self-Correcting Masked Diffusion Language Models
par: Yu, Longxuan, et autres
Publié: (2026)

MaskSearch: A Universal Pre-Training Framework to Enhance Agentic Search Capability
par: Wu, Weiqi, et autres
Publié: (2025)

Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis
par: Boughorbel, Sabri, et autres
Publié: (2024)

AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality
par: Qing, Peijun, et autres
Publié: (2024)

Structured First-Layer Initialization Pre-Training Techniques to Accelerate Training Process Based on $\varepsilon$-Rank
par: Tang, Tao, et autres
Publié: (2025)

MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphs
par: Yu, Xingtong, et autres
Publié: (2023)

SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values
par: Sun, Chengwei, et autres
Publié: (2024)

SPECTRUM: Speaker-Enhanced Pre-Training for Long Dialogue Summarization
par: Cho, Sangwoo, et autres
Publié: (2024)

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
par: Guo, Xu, et autres
Publié: (2026)

Unifying Structured Data as Graph for Data-to-Text Pre-Training
par: Li, Shujie, et autres
Publié: (2024)

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models
par: Lu, Yuheng, et autres
Publié: (2024)

How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation?
par: Uppaal, Rheeya, et autres
Publié: (2024)

Chain of Methodologies: Scaling Test Time Computation without Training
par: Liu, Cong, et autres
Publié: (2025)

Automating Legal Interpretation with LLMs: Retrieval, Generation, and Evaluation
par: Luo, Kangcheng, et autres
Publié: (2025)

ILT-Iterative LoRA Training through Focus-Feedback-Fix for Multilingual Speech Recognition
par: Meng, Qingliang, et autres
Publié: (2025)

Domain-Adaptive Continued Pre-Training of Small Language Models
par: Faroz, Salman
Publié: (2025)

How Do Large Language Models Learn Concepts During Continual Pre-Training?
par: Yao, Barry Menglong, et autres
Publié: (2026)

D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
par: Que, Haoran, et autres
Publié: (2024)

Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injection
par: Chen, Tianxiang, et autres
Publié: (2024)

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
par: Yu, Fangxu, et autres
Publié: (2024)

Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers
par: Pan, Haowen, et autres
Publié: (2023)

Integrating Pre-Trained Language Model with Physical Layer Communications
par: Lee, Ju-Hyung, et autres
Publié: (2024)

Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training
par: Baroian, Andrei, et autres
Publié: (2025)

A Study on Hidden Layer Distillation for Large Language Model Pre-Training
par: Guigon, Maxime, et autres
Publié: (2026)

From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes
par: Goriely, Zébulon, et autres
Publié: (2024)

Rethinking Data Synthesis: A Teacher Model Training Recipe with Interpretation
par: Chen, Yifang, et autres
Publié: (2024)

$ρ$-$\texttt{EOS}$: Training-free Bidirectional Variable-Length Control for Masked Diffusion LLMs
par: Yang, Jingyi, et autres
Publié: (2026)