Mid-Training of Large Language Models: A Survey
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911197684563968 |
|---|---|
| author | Mo, Kaixiang Shi, Yuxin Weng, Weiwei Zhou, Zhiqiang Liu, Shuman Zhang, Haibo Zeng, Anxiang |
| author_facet | Mo, Kaixiang Shi, Yuxin Weng, Weiwei Zhou, Zhiqiang Liu, Shuman Zhang, Haibo Zeng, Anxiang |
| contents | Large language models (LLMs) are typically developed through large-scale pre-training followed by task-specific fine-tuning. Recent advances highlight the importance of an intermediate mid-training stage, where models undergo multiple annealing-style phases that refine data quality, adapt optimization schedules, and extend context length. This stage mitigates diminishing returns from noisy tokens, stabilizes convergence, and expands model capability in late training. Its effectiveness can be explained through gradient noise scale, the information bottleneck, and curriculum learning, which together promote generalization and abstraction. Despite widespread use in state-of-the-art systems, there has been no prior survey of mid-training as a unified paradigm. We introduce the first taxonomy of LLM mid-training spanning data distribution, learning-rate scheduling, and long-context extension. We distill practical insights, compile evaluation benchmarks, and report gains to enable structured comparisons across models. We also identify open challenges and propose avenues for future research and practice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06826 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mid-Training of Large Language Models: A Survey Mo, Kaixiang Shi, Yuxin Weng, Weiwei Zhou, Zhiqiang Liu, Shuman Zhang, Haibo Zeng, Anxiang Computation and Language Large language models (LLMs) are typically developed through large-scale pre-training followed by task-specific fine-tuning. Recent advances highlight the importance of an intermediate mid-training stage, where models undergo multiple annealing-style phases that refine data quality, adapt optimization schedules, and extend context length. This stage mitigates diminishing returns from noisy tokens, stabilizes convergence, and expands model capability in late training. Its effectiveness can be explained through gradient noise scale, the information bottleneck, and curriculum learning, which together promote generalization and abstraction. Despite widespread use in state-of-the-art systems, there has been no prior survey of mid-training as a unified paradigm. We introduce the first taxonomy of LLM mid-training spanning data distribution, learning-rate scheduling, and long-context extension. We distill practical insights, compile evaluation benchmarks, and report gains to enable structured comparisons across models. We also identify open challenges and propose avenues for future research and practice. |
| title | Mid-Training of Large Language Models: A Survey |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.06826 |