Scaling with Collapse: Efficient and Predictable Training of LLM Families

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Bergsma, Shane, Zhang, Bin Claire, Dey, Nolan, Muhammad, Shaheer, Gosal, Gurpreet, Hestness, Joel
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
by: Bergsma, Shane, et al.
Published: (2025)

Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
by: Bergsma, Shane, et al.
Published: (2025)

Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
by: Bergsma, Shane, et al.
Published: (2025)

PTPP-Aware Adaptation Scaling Laws: Predicting Domain-Adaptation Performance at Unseen Pre-Training Budgets
by: Goffinet, Etienne, et al.
Published: (2025)

Sparse maximal update parameterization: A holistic approach to sparse training dynamics
by: Dey, Nolan, et al.
Published: (2024)

Don't be lazy: CompleteP enables compute-efficient deep transformers
by: Dey, Nolan, et al.
Published: (2025)

Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers
by: Gray, Gavia, et al.
Published: (2024)

CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
by: Garg, Anisha, et al.
Published: (2025)

SABER: Switchable and Balanced Training for Efficient LLM Reasoning
by: Zhao, Kai, et al.
Published: (2025)

OpenELM: An Efficient Language Model Family with Open Training and Inference Framework
by: Mehta, Sachin, et al.
Published: (2024)

Memory-Efficient LLM Training with Online Subspace Descent
by: Liang, Kaizhao, et al.
Published: (2024)

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
by: Xiao, Chaojun, et al.
Published: (2024)

POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation
by: Qiu, Zeju, et al.
Published: (2026)

Escaping Collapse: The Strength of Weak Data for Large Language Model Training
by: Amin, Kareem, et al.
Published: (2025)

A Tale of Tails: Model Collapse as a Change of Scaling Laws
by: Dohmatob, Elvis, et al.
Published: (2024)

ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates
by: Yang, Ling, et al.
Published: (2025)

Collapse of Irrelevant Representations (CIR) Ensures Robust and Non-Disruptive LLM Unlearning
by: Sondej, Filip, et al.
Published: (2025)

HAPO: Training Language Models to Reason Concisely via History-Aware Policy Optimization
by: Huang, Chengyu, et al.
Published: (2025)

How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse
by: Seddik, Mohamed El Amine, et al.
Published: (2024)

Theoretical Foundations of Scaling Law in Familial Models
by: Song, Huan, et al.
Published: (2025)

Litespark Technical Report: High-Throughput, Energy-Efficient LLM Training Framework
by: Dade, Nii Osae Osae, et al.
Published: (2025)

Muon is Scalable for LLM Training
by: Liu, Jingyuan, et al.
Published: (2025)

Survival Meets Classification: A Novel Framework for Early Risk Prediction Models of Chronic Diseases
by: Khan, Shaheer Ahmad, et al.
Published: (2026)

From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization
by: Zhou, Chenxi, et al.
Published: (2026)

MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling
by: Ding, Ning, et al.
Published: (2026)

R$^2$PO: Decoupling Training Trajectories from Inference Responses for LLM Reasoning
by: Wang, Jingchu, et al.
Published: (2026)

Better LLM Reasoning via Dual-Play
by: Zhang, Zhengxin, et al.
Published: (2025)

Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Model
by: Hajimolahoseini, Habib, et al.
Published: (2024)

TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scaling
by: Lin, Weizhe, et al.
Published: (2025)

TSO: Self-Training with Scaled Preference Optimization
by: Chen, Kaihui, et al.
Published: (2024)

PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches
by: Khan, Rana Muhammad Shahroz, et al.
Published: (2024)

EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
by: Chen, Mengzhao, et al.
Published: (2024)

Training Proactive and Personalized LLM Agents
by: Sun, Weiwei, et al.
Published: (2025)

Atom of Thoughts for Markov LLM Test-Time Scaling
by: Teng, Fengwei, et al.
Published: (2025)

CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation
by: Fawi, Muhammad
Published: (2024)

AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs
by: Kang, Feiyang, et al.
Published: (2024)

BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
by: Huang, Wei, et al.
Published: (2024)

Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
by: Zhang, Ruiqi, et al.
Published: (2024)

KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training
by: Kumaresan, Ramchand
Published: (2026)

P$^2$ Law: Scaling Law for Post-Training After Model Pruning
by: Chen, Xiaodong, et al.
Published: (2024)