Copertina

Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Merrill, William, Arora, Shane, Groeneveld, Dirk, Hajishirzi, Hannaneh
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference
di: Zhao, Bowen, et al.
Pubblicazione: (2024)

Scaling Law for Language Models Training Considering Batch Size
di: Shuai, Xian, et al.
Pubblicazione: (2024)

Convergence Bound and Critical Batch Size of Muon Optimizer
di: Sato, Naoki, et al.
Pubblicazione: (2025)

Revisiting LARS for Large Batch Training Generalization of Neural Networks
di: Do, Khoi, et al.
Pubblicazione: (2023)

Full-Graph vs. Mini-Batch Training: Comprehensive Analysis from a Batch Size and Fan-Out Size Perspective
di: Liu, Mengfan, et al.
Pubblicazione: (2026)

Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism
di: Lau, Tim Tsz-Kit, et al.
Pubblicazione: (2024)

AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size
di: Ostroukhov, Petr, et al.
Pubblicazione: (2024)

A Simple and Efficient Approach to Batch Bayesian Optimization
di: Zhan, Dawei, et al.
Pubblicazione: (2024)

How Does Critical Batch Size Scale in Pre-training?
di: Zhang, Hanlin, et al.
Pubblicazione: (2024)

DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptation
di: Chen, Yuen, et al.
Pubblicazione: (2025)

Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful
di: Marek, Martin, et al.
Pubblicazione: (2025)

Learning to Detect Language Model Training Data via Active Reconstruction
di: Yin, Junjie Oscar, et al.
Pubblicazione: (2026)

Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training
di: Zhao, Shen-Yi, et al.
Pubblicazione: (2020)

Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performance
di: Piao, XinYu, et al.
Pubblicazione: (2021)

Diversified Batch Selection for Training Acceleration
di: Hong, Feng, et al.
Pubblicazione: (2024)

EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Trees
di: Zeng, Zhiyuan, et al.
Pubblicazione: (2025)

Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model Editing with Llama-3
di: Yoon, Junsang, et al.
Pubblicazione: (2024)

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
di: Bergsma, Shane, et al.
Pubblicazione: (2025)

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
di: Meterez, Alexandru, et al.
Pubblicazione: (2025)

Smaller Batches, Bigger Gains? Investigating the Impact of Batch Sizes on Reinforcement Learning Based Real-World Production Scheduling
di: Müller, Arthur, et al.
Pubblicazione: (2024)

Husky: A Unified, Open-Source Language Agent for Multi-Step Reasoning
di: Kim, Joongwon, et al.
Pubblicazione: (2024)

Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling
di: Li, Shuaipeng, et al.
Pubblicazione: (2024)

Collaborative Batch Size Optimization for Federated Learning
di: Geimer, Arno, et al.
Pubblicazione: (2025)

On the Role of Batch Size in Stochastic Conditional Gradient Methods
di: Islamov, Rustem, et al.
Pubblicazione: (2026)

Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
di: Chen, Tong, et al.
Pubblicazione: (2025)

An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms
di: Wang, Li, et al.
Pubblicazione: (2025)

Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Training
di: Tyagi, Sahil, et al.
Pubblicazione: (2026)

Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentum
di: Kamo, Keisuke, et al.
Pubblicazione: (2025)

Riemannian Batch Normalization: A Gyro Approach
di: Chen, Ziheng, et al.
Pubblicazione: (2025)

Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation
di: Räisä, Ossi, et al.
Pubblicazione: (2024)

Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA
di: Lee, Sangyoon, et al.
Pubblicazione: (2026)

Robust Batched Bandits
di: Guo, Yunwen, et al.
Pubblicazione: (2025)

Supervised Batch Normalization
di: Faye, Bilal, et al.
Pubblicazione: (2024)

Batch Bayesian Active Learning with Partial Batch Label Sampling
di: Hu, Kangping, et al.
Pubblicazione: (2025)

Adaptive Batch Sizes for Active Learning A Probabilistic Numerics Approach
di: Adachi, Masaki, et al.
Pubblicazione: (2023)

BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching
di: Zheng, Zhen, et al.
Pubblicazione: (2024)

Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantization
di: Zhang, Xinyuan, et al.
Pubblicazione: (2024)

Decoding-Time Language Model Alignment with Multiple Objectives
di: Shi, Ruizhe, et al.
Pubblicazione: (2024)

Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size
di: Oowada, Kanata, et al.
Pubblicazione: (2025)

Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging
di: Morrison, Jacob, et al.
Pubblicazione: (2024)