Imagen de Portada

How to Set the Batch Size for Large-Scale Pre-training?

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Zhou, Yunhua, Huang, Junhao, Xing, Shuhao, Zhang, Yechen, Peng, Runyu, Guo, Qiping, Qiu, Xipeng
Formato:	Preprint
Publicado:	2026
Materias:	Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

How to Set the Learning Rate for Large-Scale Pre-training?
por: Zhou, Yunhua, et al.
Publicado: (2026)

Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning
por: Zhang, Yechen, et al.
Publicado: (2026)

How Does Critical Batch Size Scale in Pre-training?
por: Zhang, Hanlin, et al.
Publicado: (2024)

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
por: Peng, Runyu, et al.
Publicado: (2026)

BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
por: Wang, Xinghao, et al.
Publicado: (2024)

Evolution of Concepts in Language Model Pre-Training
por: Ge, Xuyang, et al.
Publicado: (2025)

Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?
por: Zeng, Zhiyuan, et al.
Publicado: (2025)

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
por: Bergsma, Shane, et al.
Publicado: (2025)

How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
por: Peng, Runyu, et al.
Publicado: (2026)

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
por: Wang, Bo, et al.
Publicado: (2025)

Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective
por: Zeng, Zhiyuan, et al.
Publicado: (2024)

How to Mitigate Overfitting in Weak-to-strong Generalization?
por: Shi, Junhao, et al.
Publicado: (2025)

World-aware Planning Narratives Enhance Large Vision-Language Model Planner
por: Shi, Junhao, et al.
Publicado: (2025)

ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models
por: Yin, Zhangyue, et al.
Publicado: (2025)

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
por: Ye, Jiasheng, et al.
Publicado: (2024)

Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models
por: Xu, Ningyu, et al.
Publicado: (2026)

How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models
por: Lv, Kangtao, et al.
Publicado: (2025)

Batch-in-Batch: a new adversarial training framework for initial perturbation and sample selection
por: Wu, Yinting, et al.
Publicado: (2024)

Aligning Instruction Tuning with Pre-training
por: Liang, Yiming, et al.
Publicado: (2025)

An Aligned Constraint Programming Model For Serial Batch Scheduling With Minimum Batch Size
por: Huertas, Jorge A., et al.
Publicado: (2025)

Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training
por: Zhang, Xinsong, et al.
Publicado: (2025)

Data-free Weight Compress and Denoise for Large Language Models
por: Peng, Runyu, et al.
Publicado: (2024)

Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editing
por: Guo, Dongliang, et al.
Publicado: (2024)

Scaling Laws of RoPE-based Extrapolation
por: Liu, Xiaoran, et al.
Publicado: (2023)

Scaling LLM Pre-training with Vocabulary Curriculum
por: Yu, Fangyuan
Publicado: (2025)

Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning
por: Fei, Zhaoye, et al.
Publicado: (2025)

Pre-training Distillation for Large Language Models: A Design Space Exploration
por: Peng, Hao, et al.
Publicado: (2024)

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE
por: Zeng, Zhiyuan, et al.
Publicado: (2024)

DataMan: Data Manager for Pre-training Large Language Models
por: Peng, Ru, et al.
Publicado: (2025)

Pre-train and Fine-tune: Recommenders as Large Models
por: Jiang, Zhenhao, et al.
Publicado: (2025)

OSF: On Pre-training and Scaling of Sleep Foundation Models
por: Shuai, Zitao, et al.
Publicado: (2026)

Scaling Laws for Pre-training Agents and World Models
por: Pearce, Tim, et al.
Publicado: (2024)

CLSP: High-Fidelity Contrastive Language-State Pre-training for Agent State Representation
por: Huang, Fuxian, et al.
Publicado: (2024)

Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent
por: Naganuma, Hiroki, et al.
Publicado: (2026)

Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models
por: Ma, Shengjie, et al.
Publicado: (2025)

Evaluating the Performance of Large Language Models on GAOKAO Benchmark
por: Zhang, Xiaotian, et al.
Publicado: (2023)

Leveraging Pre-trained Large Language Models with Refined Prompting for Online Task and Motion Planning
por: Guo, Huihui, et al.
Publicado: (2025)

Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets
por: Jiang, Guangqi, et al.
Publicado: (2024)

Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making
por: Ruan, Jingqing, et al.
Publicado: (2023)

BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching
por: Zheng, Zhen, et al.
Publicado: (2024)