Copertina

Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Dremov, Aleksandr, Hägele, Alexander, Kosson, Atli, Jaggi, Martin
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training
di: Kosson, Atli, et al.
Pubblicazione: (2024)

Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
di: Kosson, Atli, et al.
Pubblicazione: (2023)

Weight Decay may matter more than muP for Learning Rate Transfer in Practice
di: Kosson, Atli, et al.
Pubblicazione: (2025)

Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
di: Hägele, Alexander, et al.
Pubblicazione: (2024)

Compute-Optimal Quantization-Aware Training
di: Dremov, Aleksandr, et al.
Pubblicazione: (2025)

Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay
di: Li, Binghui, et al.
Pubblicazione: (2026)

Optimal Linear Decay Learning Rate Schedules and Further Refinements
di: Defazio, Aaron, et al.
Pubblicazione: (2023)

Efficient Generative Model Training via Embedded Representation Warmup
di: Liu, Deyuan, et al.
Pubblicazione: (2025)

Data Warmup: Complexity-Aware Curricula for Efficient Diffusion Training
di: Lin, Jinhong, et al.
Pubblicazione: (2026)

Test-Time Warmup for Multimodal Large Language Models
di: Rajaneesh, Nikita, et al.
Pubblicazione: (2025)

Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformers
di: Belloni, Annalisa, et al.
Pubblicazione: (2026)

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
di: Meterez, Alexandru, et al.
Pubblicazione: (2025)

Hierarchical Mixture-of-Experts with Two-Stage Optimization
di: Molodtsov, Gleb, et al.
Pubblicazione: (2026)

Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
di: Wen, Kaiyue, et al.
Pubblicazione: (2024)

ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
di: Defazio, Aaron
Pubblicazione: (2026)

Cyclical Log Annealing as a Learning Rate Scheduler
di: Naveen, Philip
Pubblicazione: (2024)

Dynamic Low-rank Approximation of Full-Matrix Preconditioner for Training Generalized Linear Models
di: Matveeva, Tatyana, et al.
Pubblicazione: (2025)

Super Level Sets and Exponential Decay: A Synergistic Approach to Stable Neural Network Training
di: Chaudhary, Jatin, et al.
Pubblicazione: (2024)

Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate
di: Xu, Huangyu, et al.
Pubblicazione: (2026)

NeuralGrok: Accelerate Grokking by Neural Gradient Transformation
di: Zhou, Xinyu, et al.
Pubblicazione: (2025)

Heterogeneous Learning Rate Scheduling for Neural Architecture Search on Long-Tailed Datasets
di: Tang, Chenxia
Pubblicazione: (2024)

Two-Stage Representation Learning for Analyzing Movement Behavior Dynamics in People Living with Dementia
di: Cui, Jin, et al.
Pubblicazione: (2025)

The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
di: Schaipp, Fabian, et al.
Pubblicazione: (2025)

DoGE: Domain Reweighting with Generalization Estimation
di: Fan, Simin, et al.
Pubblicazione: (2023)

Towards an empirical understanding of MoE design choices
di: Fan, Dongyang, et al.
Pubblicazione: (2024)

Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler
di: Shen, Yikang, et al.
Pubblicazione: (2024)

Load-Aware Training Scheduling for Model Circulation-based Decentralized Federated Learning
di: Kainuma, Haruki, et al.
Pubblicazione: (2025)

Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages
di: Ma, Guozheng, et al.
Pubblicazione: (2023)

Unsupervised Physics-Informed Operator Learning through Multi-Stage Curriculum Training
di: Marcandelli, Paolo, et al.
Pubblicazione: (2026)

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
di: Meterez, Alexandru, et al.
Pubblicazione: (2026)

Dynamic Temperature Scheduler for Knowledge Distillation
di: Islam, Sibgat Ul, et al.
Pubblicazione: (2025)

Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning
di: Qiu, Haomiao, et al.
Pubblicazione: (2025)

Analyzing Patient Daily Movement Behavior Dynamics Using Two-Stage Encoding Model
di: Cui, Jin, et al.
Pubblicazione: (2025)

A theoretical framework for self-supervised contrastive learning for continuous dependent data
di: Marusov, Alexander, et al.
Pubblicazione: (2025)

$α$-LoRA: Effective Fine-Tuning via Base Model Rescaling
di: Firdoussi, Aymane El, et al.
Pubblicazione: (2025)

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
di: Luo, Kairong, et al.
Pubblicazione: (2025)

Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs
di: Bossy, Thierry, et al.
Pubblicazione: (2025)

Beyond Training: Optimizing Reinforcement Learning Based Job Shop Scheduling Through Adaptive Action Sampling
di: de Puiseau, Constantin Waubert, et al.
Pubblicazione: (2024)

Self-Abstraction Learning for Effective and Stable Training of Deep Neural Networks
di: Cho, Wonyong, et al.
Pubblicazione: (2026)

A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules
di: Luo, Kairong, et al.
Pubblicazione: (2025)