Variance Control via Weight Rescaling in LLM Pre-training

Documenti analoghi

• ZClip: Adaptive Spike Mitigation for LLM Pre-Training
  di: Kumar, Abhay, et al.
  Pubblicazione: (2025)
• A Refined Analysis of Massive Activations in LLMs
  di: Owen, Louis, et al.
  Pubblicazione: (2025)
• Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
  di: Bergsma, Shane, et al.
  Pubblicazione: (2025)
• High-Layer Attention Pruning with Rescaling
  di: Liu, Songtao, et al.
  Pubblicazione: (2025)
• nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales
  di: Yao, Yiqun, et al.
  Pubblicazione: (2023)