Copertina

GradPower: Powering Gradients for Faster Language Model Pre-Training

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Wang, Jinbo, Wang, Mingze, Zhang, Jiaqi, Wang, Wei, Pei, Peng, Cai, Xunliang, E, Weinan, Wu, Lei
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Optimization and Control
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
di: Wang, Jinbo, et al.
Pubblicazione: (2025)

Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws
di: Wang, Jinbo, et al.
Pubblicazione: (2026)

How Transformers Get Rich: Approximation and Dynamics Analysis
di: Wang, Mingze, et al.
Pubblicazione: (2024)

Improving Generalization and Convergence by Enhancing Implicit Regularization
di: Wang, Mingze, et al.
Pubblicazione: (2024)

Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent
di: Ziyin, Liu, et al.
Pubblicazione: (2024)

Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescaling
di: Wang, Mingze, et al.
Pubblicazione: (2023)

MARS: Unleashing the Power of Variance Reduction for Training Large Models
di: Yuan, Huizhuo, et al.
Pubblicazione: (2024)

AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithm
di: Bojovic, Matia, et al.
Pubblicazione: (2026)

AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
di: Lau, Tim Tsz-Kit, et al.
Pubblicazione: (2024)

From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees
di: Xie, Shengping, et al.
Pubblicazione: (2025)

PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective
di: Lau, Tim Tsz-Kit, et al.
Pubblicazione: (2025)

Faster Gradient Methods for Highly-Smooth Stochastic Bilevel Optimization
di: Chen, Lesi, et al.
Pubblicazione: (2025)

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation
di: Mishkin, Aaron, et al.
Pubblicazione: (2024)

Faster Gradient-Free Algorithms for Nonsmooth Nonconvex Stochastic Optimization
di: Chen, Lesi, et al.
Pubblicazione: (2023)

Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad
di: Liu, Zijian
Pubblicazione: (2026)

GNMR: Runtime Stability Control for Low-Precision Large Language Model Training
di: Kong, Boao, et al.
Pubblicazione: (2026)

A Variance-Reduced Stochastic Gradient Tracking Algorithm for Decentralized Optimization with Orthogonality Constraints
di: Wang, Lei, et al.
Pubblicazione: (2022)

Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size
di: Oowada, Kanata, et al.
Pubblicazione: (2025)

AdaGrad under Anisotropic Smoothness
di: Liu, Yuxing, et al.
Pubblicazione: (2024)

Armijo Line-search Can Make (Stochastic) Gradient Descent Provably Faster
di: Vaswani, Sharan, et al.
Pubblicazione: (2025)

Sample Robust Scheduling of Electricity-Gas Systems Under Wind Power Uncertainty
di: Liu, Rong-Peng, et al.
Pubblicazione: (2023)

Revisiting Convergence of AdaGrad with Relaxed Assumptions
di: Hong, Yusu, et al.
Pubblicazione: (2024)

Provably Faster Gradient Descent via Long Steps
di: Grimmer, Benjamin
Pubblicazione: (2023)

Faster Reinforcement Learning by Freezing Slow States
di: Wang, Yijia, et al.
Pubblicazione: (2023)

Expressive Power of Graph Neural Networks for (Mixed-Integer) Quadratic Programs
di: Chen, Ziang, et al.
Pubblicazione: (2024)

AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates
di: Zhang, Minxin, et al.
Pubblicazione: (2025)

Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networks
di: Xu, Xianliang, et al.
Pubblicazione: (2024)

Online Planning of Power Flows for Power Systems Against Bushfires Using Spatial Context
di: Xu, Jianyu, et al.
Pubblicazione: (2024)

Model-Free Output Feedback Stabilization via Policy Gradient Methods
di: Zhang, Ankang, et al.
Pubblicazione: (2026)

GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity
di: Maranjyan, Artavazd, et al.
Pubblicazione: (2022)

Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations
di: Heredia, Carlos
Pubblicazione: (2024)

Faster Convergence of Local SGD for Over-Parameterized Models
di: Qin, Tiancheng, et al.
Pubblicazione: (2022)

AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size
di: Ostroukhov, Petr, et al.
Pubblicazione: (2024)

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks
di: Liu, Xin, et al.
Pubblicazione: (2022)

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
di: Wang, Mingze, et al.
Pubblicazione: (2024)

Decentralized Stability-Constrained Optimal Power Flow for Inverter-Based Power Systems
di: Wang, Shigeng, et al.
Pubblicazione: (2026)

Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
di: Chezhegov, Savelii, et al.
Pubblicazione: (2024)

Faster Acceleration for Steepest Descent
di: Bai, Cedar Site, et al.
Pubblicazione: (2024)

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
di: Xie, Xingyu, et al.
Pubblicazione: (2022)

Power-Constrained Policy Gradient Methods for LQR
di: Verma, Ashwin, et al.
Pubblicazione: (2025)