Imagen de Portada

The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Wang, Jinbo, Wang, Mingze, Zhou, Zhanpeng, Yan, Junchi, E, Weinan, Wu, Lei
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning Artificial Intelligence Optimization and Control
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

GradPower: Powering Gradients for Faster Language Model Pre-Training
por: Wang, Jinbo, et al.
Publicado: (2025)

How Transformers Get Rich: Approximation and Dynamics Analysis
por: Wang, Mingze, et al.
Publicado: (2024)

Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws
por: Wang, Jinbo, et al.
Publicado: (2026)

Benchmarking PtO and PnO Methods in the Predictive Combinatorial Optimization Regime
por: Geng, Haoyu, et al.
Publicado: (2023)

Double-Bounded Optimal Transport for Advanced Clustering and Classification
por: Shi, Liangliang, et al.
Publicado: (2024)

On the Duality Between Sharpness-Aware Minimization and Adversarial Training
por: Zhang, Yihao, et al.
Publicado: (2024)

Improving Generalization and Convergence by Enhancing Implicit Regularization
por: Wang, Mingze, et al.
Publicado: (2024)

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks
por: Liu, Xin, et al.
Publicado: (2022)

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
por: Meterez, Alexandru, et al.
Publicado: (2025)

Optimal Control Operator Perspective and a Neural Adaptive Spectral Method
por: Feng, Mingquan, et al.
Publicado: (2024)

Accelerating RLHF Training with Reward Variance Increase
por: Yang, Zonglin, et al.
Publicado: (2025)

YuriiFormer: A Suite of Nesterov-Accelerated Transformers
por: Zimin, Aleksandr, et al.
Publicado: (2026)

Training Infinitely Deep and Wide Transformers
por: Barboni, Raphaël, et al.
Publicado: (2026)

SGD at the Edge of Stability: The Stochastic Sharpness Gap
por: Liao, Fangshuo, et al.
Publicado: (2026)

Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training
por: Song, Minhak, et al.
Publicado: (2025)

StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
por: Yu, Dingzhi, et al.
Publicado: (2026)

Provable Acceleration for Diffusion Models under Minimal Assumptions
por: Li, Gen, et al.
Publicado: (2024)

LinSATNet: The Positive Linear Satisfiability Neural Networks
por: Wang, Runzhong, et al.
Publicado: (2024)

PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Training
por: Yang, Shenghao, et al.
Publicado: (2026)

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
por: Chen, Siyu, et al.
Publicado: (2024)

How Does Critical Batch Size Scale in Pre-training?
por: Zhang, Hanlin, et al.
Publicado: (2024)

Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regime
por: Wang, Yuqing, et al.
Publicado: (2025)

Solving Functional Optimization with Deep Networks and Variational Principles
por: Kamtue, Kawisorn, et al.
Publicado: (2024)

Understanding Sampler Stochasticity in Training Diffusion Models for RLHF
por: Sheng, Jiayuan, et al.
Publicado: (2025)

How Well Can Transformers Emulate In-context Newton's Method?
por: Giannou, Angeliki, et al.
Publicado: (2024)

Demystifying Manifold Constraints in LLM Pre-training
por: An, Kang, et al.
Publicado: (2026)

Physics-informed Neural-operator Predictive Control for Drag Reduction in Turbulent Flows
por: Zhao, Zelin, et al.
Publicado: (2025)

PID Accelerated Temporal Difference Algorithms
por: Bedaywi, Mark, et al.
Publicado: (2024)

Unsupervised Training of Diffusion Models for Feasible Solution Generation in Neural Combinatorial Optimization
por: Hong, Seong-Hyun, et al.
Publicado: (2024)

Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescaling
por: Wang, Mingze, et al.
Publicado: (2023)

One-Layer Transformer Provably Learns One-Nearest Neighbor In Context
por: Li, Zihao, et al.
Publicado: (2024)

One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer
por: Shen, Jucheng, et al.
Publicado: (2026)

LLM Serving Optimization with Variable Prefill and Decode Lengths
por: Wang, Meixuan, et al.
Publicado: (2025)

From Soliloquy to Agora: Memory-Enhanced LLM Agents with Decentralized Debate for Optimization Modeling
por: Lin, Jianghao, et al.
Publicado: (2026)

Accelerating Cutting-Plane Algorithms via Reinforcement Learning Surrogates
por: Mana, Kyle, et al.
Publicado: (2023)

Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers
por: Lau, Tim Tsz-Kit, et al.
Publicado: (2026)

Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
por: Hao, Yongchang, et al.
Publicado: (2026)

Solving General Natural-Language-Description Optimization Problems with Large Language Models
por: Zhang, Jihai, et al.
Publicado: (2024)

Anytime Training with Schedule-Free Spectral Optimization
por: Apte, Anuj, et al.
Publicado: (2026)

Stability of Transformers under Layer Normalization
por: Kan, Kelvin, et al.
Publicado: (2025)