Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

Documenti analoghi

• Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler
  di: Dremov, Aleksandr, et al.
  Pubblicazione: (2025)
• Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training
  di: Kosson, Atli, et al.
  Pubblicazione: (2024)
• Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
  di: Kosson, Atli, et al.
  Pubblicazione: (2023)
• Weight Decay may matter more than muP for Learning Rate Transfer in Practice
  di: Kosson, Atli, et al.
  Pubblicazione: (2025)
• Towards Fully FP8 GEMM LLM Training at Scale
  di: Hernández-Cano, Alejandro, et al.
  Pubblicazione: (2025)