Image de couverture de livre

Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Merrill, William, Arora, Shane, Groeneveld, Dirk, Hajishirzi, Hannaneh
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference
par: Zhao, Bowen, et autres
Publié: (2024)

Scaling Law for Language Models Training Considering Batch Size
par: Shuai, Xian, et autres
Publié: (2024)

Convergence Bound and Critical Batch Size of Muon Optimizer
par: Sato, Naoki, et autres
Publié: (2025)

Revisiting LARS for Large Batch Training Generalization of Neural Networks
par: Do, Khoi, et autres
Publié: (2023)

Full-Graph vs. Mini-Batch Training: Comprehensive Analysis from a Batch Size and Fan-Out Size Perspective
par: Liu, Mengfan, et autres
Publié: (2026)

Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism
par: Lau, Tim Tsz-Kit, et autres
Publié: (2024)

AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size
par: Ostroukhov, Petr, et autres
Publié: (2024)

A Simple and Efficient Approach to Batch Bayesian Optimization
par: Zhan, Dawei, et autres
Publié: (2024)

How Does Critical Batch Size Scale in Pre-training?
par: Zhang, Hanlin, et autres
Publié: (2024)

DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptation
par: Chen, Yuen, et autres
Publié: (2025)

Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful
par: Marek, Martin, et autres
Publié: (2025)

Learning to Detect Language Model Training Data via Active Reconstruction
par: Yin, Junjie Oscar, et autres
Publié: (2026)

Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training
par: Zhao, Shen-Yi, et autres
Publié: (2020)

Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performance
par: Piao, XinYu, et autres
Publié: (2021)

Diversified Batch Selection for Training Acceleration
par: Hong, Feng, et autres
Publié: (2024)

EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Trees
par: Zeng, Zhiyuan, et autres
Publié: (2025)

Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model Editing with Llama-3
par: Yoon, Junsang, et autres
Publié: (2024)

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
par: Bergsma, Shane, et autres
Publié: (2025)

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
par: Meterez, Alexandru, et autres
Publié: (2025)

Smaller Batches, Bigger Gains? Investigating the Impact of Batch Sizes on Reinforcement Learning Based Real-World Production Scheduling
par: Müller, Arthur, et autres
Publié: (2024)

Husky: A Unified, Open-Source Language Agent for Multi-Step Reasoning
par: Kim, Joongwon, et autres
Publié: (2024)

Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling
par: Li, Shuaipeng, et autres
Publié: (2024)

Collaborative Batch Size Optimization for Federated Learning
par: Geimer, Arno, et autres
Publié: (2025)

On the Role of Batch Size in Stochastic Conditional Gradient Methods
par: Islamov, Rustem, et autres
Publié: (2026)

Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
par: Chen, Tong, et autres
Publié: (2025)

An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms
par: Wang, Li, et autres
Publié: (2025)

Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Training
par: Tyagi, Sahil, et autres
Publié: (2026)

Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentum
par: Kamo, Keisuke, et autres
Publié: (2025)

Riemannian Batch Normalization: A Gyro Approach
par: Chen, Ziheng, et autres
Publié: (2025)

Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation
par: Räisä, Ossi, et autres
Publié: (2024)

Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA
par: Lee, Sangyoon, et autres
Publié: (2026)

Robust Batched Bandits
par: Guo, Yunwen, et autres
Publié: (2025)

Supervised Batch Normalization
par: Faye, Bilal, et autres
Publié: (2024)

Batch Bayesian Active Learning with Partial Batch Label Sampling
par: Hu, Kangping, et autres
Publié: (2025)

Adaptive Batch Sizes for Active Learning A Probabilistic Numerics Approach
par: Adachi, Masaki, et autres
Publié: (2023)

BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching
par: Zheng, Zhen, et autres
Publié: (2024)

Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantization
par: Zhang, Xinyuan, et autres
Publié: (2024)

Decoding-Time Language Model Alignment with Multiple Objectives
par: Shi, Ruizhe, et autres
Publié: (2024)

Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size
par: Oowada, Kanata, et autres
Publié: (2025)

Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging
par: Morrison, Jacob, et autres
Publié: (2024)