Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training

Documenti analoghi

• APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference
  di: Zhao, Bowen, et al.
  Pubblicazione: (2024)
• Scaling Law for Language Models Training Considering Batch Size
  di: Shuai, Xian, et al.
  Pubblicazione: (2024)
• Convergence Bound and Critical Batch Size of Muon Optimizer
  di: Sato, Naoki, et al.
  Pubblicazione: (2025)
• Revisiting LARS for Large Batch Training Generalization of Neural Networks
  di: Do, Khoi, et al.
  Pubblicazione: (2023)
• Full-Graph vs. Mini-Batch Training: Comprehensive Analysis from a Batch Size and Fan-Out Size Perspective
  di: Liu, Mengfan, et al.
  Pubblicazione: (2026)