Cover Image

Exploring Scaling Laws for Local SGD in Large Language Model Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	He, Qiaozhi, Zhuang, Xiaomin, Wu, Zhihua
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Code Comparison Tuning for Code Large Language Models
by: Jiang, Yufan, et al.
Published: (2024)

Asynchronous Local-SGD Training for Language Modeling
by: Liu, Bo, et al.
Published: (2024)

ChuXin: 1.6B Technical Report
by: Zhuang, Xiaomin, et al.
Published: (2024)

Scaling Laws for Post Training Quantized Large Language Models
by: Xu, Zifei, et al.
Published: (2024)

Scaling Law for Language Models Training Considering Batch Size
by: Shuai, Xian, et al.
Published: (2024)

Scaling Laws for Discriminative Classification in Large Language Models
by: Wyatte, Dean, et al.
Published: (2024)

Scaling Laws for Multilingual Language Models
by: He, Yifei, et al.
Published: (2024)

Performance Law of Large Language Models
by: Wu, Chuhan, et al.
Published: (2024)

Efficient LLM Inference with Kcache
by: He, Qiaozhi, et al.
Published: (2024)

RecycleGPT: An Autoregressive Language Model with Recyclable Module
by: Jiang, Yufan, et al.
Published: (2023)

Scaling Laws for Downstream Task Performance of Large Language Models
by: Isik, Berivan, et al.
Published: (2024)

Task-Stratified Knowledge Scaling Laws for Post-Training Quantized Large Language Models
by: Zhou, Chenxi, et al.
Published: (2025)

Parallel Scaling Law for Language Models
by: Chen, Mouxiang, et al.
Published: (2025)

Scaling Law for Quantization-Aware Training
by: Chen, Mengzhao, et al.
Published: (2025)

Uncovering Scaling Laws for Large Language Models via Inverse Problems
by: Verma, Arun, et al.
Published: (2025)

SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models
by: Huo, Yifu, et al.
Published: (2026)

Exploring Scaling Laws for EHR Foundation Models
by: Zhang, Sheng, et al.
Published: (2025)

Prescriptive Scaling Laws for Data Constrained Training
by: Lovelace, Justin, et al.
Published: (2026)

Scaling Laws for Upcycling Mixture-of-Experts Language Models
by: Liew, Seng Pei, et al.
Published: (2025)

Provable Scaling Laws for the Test-Time Compute of Large Language Models
by: Chen, Yanxi, et al.
Published: (2024)

Exploring Cross-Client Memorization of Training Data in Large Language Models for Federated Learning
by: Udsa, Tinnakit, et al.
Published: (2025)

A Law of Next-Token Prediction in Large Language Models
by: He, Hangfeng, et al.
Published: (2024)

P$^2$ Law: Scaling Law for Post-Training After Model Pruning
by: Chen, Xiaodong, et al.
Published: (2024)

Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes On
by: Zeng, Liang, et al.
Published: (2024)

Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
by: Charles, Zachary, et al.
Published: (2025)

Selecting Large Language Model to Fine-tune via Rectified Scaling Law
by: Lin, Haowei, et al.
Published: (2024)

Linear Dynamics in the RLVR Training of Large Language Models
by: Wang, Tianle, et al.
Published: (2026)

Scaling Laws for Floating Point Quantization Training
by: Sun, Xingwu, et al.
Published: (2025)

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
by: Sardana, Nikhil, et al.
Published: (2023)

Scaling Laws for Forgetting When Fine-Tuning Large Language Models
by: Kalajdzievski, Damjan
Published: (2024)

Can Language Models Discover Scaling Laws?
by: Lin, Haowei, et al.
Published: (2025)

Massive Activations in Large Language Models
by: Sun, Mingjie, et al.
Published: (2024)

AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training
by: Li, Zhiyuan, et al.
Published: (2026)

Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
by: Krajewski, Jakub, et al.
Published: (2025)

Observational Scaling Laws and the Predictability of Language Model Performance
by: Ruan, Yangjun, et al.
Published: (2024)

Relative-Based Scaling Law for Neural Language Models
by: Yue, Baoqing, et al.
Published: (2025)

Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing
by: Wu, Chen, et al.
Published: (2025)

LawLLM: Law Large Language Model for the US Legal System
by: Shu, Dong, et al.
Published: (2024)

Training Language Models with Language Feedback at Scale
by: Scheurer, Jérémy, et al.
Published: (2023)

Self-Training Large Language Models with Confident Reasoning
by: Jang, Hyosoon, et al.
Published: (2025)