Image de couverture de livre

Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Hägele, Alexander, Bakouch, Elie, Kosson, Atli, Allal, Loubna Ben, Von Werra, Leandro, Jaggi, Martin
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler
par: Dremov, Aleksandr, et autres
Publié: (2025)

Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training
par: Kosson, Atli, et autres
Publié: (2024)

Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
par: Kosson, Atli, et autres
Publié: (2023)

Weight Decay may matter more than muP for Learning Rate Transfer in Practice
par: Kosson, Atli, et autres
Publié: (2025)

Towards Fully FP8 GEMM LLM Training at Scale
par: Hernández-Cano, Alejandro, et autres
Publié: (2025)

Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Training
par: Harma, Simla Burcu, et autres
Publié: (2022)

Navigating Scaling Laws: Compute Optimality in Adaptive Model Training
par: Anagnostidis, Sotiris, et autres
Publié: (2023)

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
par: Niklaus, Joel, et autres
Publié: (2026)

On Neural Scaling Laws for Weather Emulation through Continual Training
par: Subramanian, Shashank, et autres
Publié: (2026)

The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
par: Schaipp, Fabian, et autres
Publié: (2025)

Scaling Law for Quantization-Aware Training
par: Chen, Mengzhao, et autres
Publié: (2025)

Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks
par: Qiu, Shikai, et autres
Publié: (2025)

Scaling Laws for Post Training Quantized Large Language Models
par: Xu, Zifei, et autres
Publié: (2024)

Prescriptive Scaling Laws for Data Constrained Training
par: Lovelace, Justin, et autres
Publié: (2026)

Compute-Optimal Quantization-Aware Training
par: Dremov, Aleksandr, et autres
Publié: (2025)

Training Transformers for Mesh-Based Simulations
par: Garnier, Paul, et autres
Publié: (2025)

Student-Informed Teacher Training
par: Messikommer, Nico, et autres
Publié: (2024)

Scaling Laws for Floating Point Quantization Training
par: Sun, Xingwu, et autres
Publié: (2025)

Test-Time Training Scaling Laws for Chemical Exploration in Drug Design
par: Thomas, Morgan, et autres
Publié: (2025)

Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models
par: Nimmaturi, Datta, et autres
Publié: (2025)

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
par: Sardana, Nikhil, et autres
Publié: (2023)

CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference
par: Mohtashami, Amirkeivan, et autres
Publié: (2023)

Scaling Law for Language Models Training Considering Batch Size
par: Shuai, Xian, et autres
Publié: (2024)

Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
par: Li, Binghui, et autres
Publié: (2026)

When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training
par: Morgan, Alexander, et autres
Publié: (2026)

Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes It
par: Zhang, Yaxiang, et autres
Publié: (2026)

Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
par: Singh, Avi, et autres
Publié: (2023)

P$^2$ Law: Scaling Law for Post-Training After Model Pruning
par: Chen, Xiaodong, et autres
Publié: (2024)

Scaling Laws for Optimal Data Mixtures
par: Shukor, Mustafa, et autres
Publié: (2025)

Exploring Scaling Laws for Local SGD in Large Language Model Training
par: He, Qiaozhi, et autres
Publié: (2024)

Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs
par: Chen, Zhengyu, et autres
Publié: (2025)

Training Compute-Optimal Protein Language Models
par: Cheng, Xingyi, et autres
Publié: (2024)

Scaling Law for Large-Scale Pre-Training Using Chaotic Time Series and Predictability in Financial Time Series
par: Takemoto, Yuki
Publié: (2025)

FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
par: Penedo, Guilherme, et autres
Publié: (2025)

Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training
par: Southworth, Ben S., et autres
Publié: (2026)

Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design
par: Alabdulmohsin, Ibrahim, et autres
Publié: (2023)

4+3 Phases of Compute-Optimal Neural Scaling Laws
par: Paquette, Elliot, et autres
Publié: (2024)

Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG
par: Han, Seungju, et autres
Publié: (2026)

A Training Framework for Optimal and Stable Training of Polynomial Neural Networks
par: Hossain, Forsad Al, et autres
Publié: (2025)

CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computation
par: Chen, Yutong, et autres
Publié: (2026)