Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training

Documenti analoghi

• Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
  di: Bergsma, Shane, et al.
  Pubblicazione: (2025)
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
  di: Bergsma, Shane, et al.
  Pubblicazione: (2025)
• Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
  di: Bergsma, Shane, et al.
  Pubblicazione: (2025)
• Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers
  di: Gray, Gavia, et al.
  Pubblicazione: (2024)
• Sparse maximal update parameterization: A holistic approach to sparse training dynamics
  di: Dey, Nolan, et al.
  Pubblicazione: (2024)