Imagen de Portada

Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Southworth, Ben S., Thomas, Stephen
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning Numerical Analysis Optimization and Control
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Faster Randomized Methods for Orthogonality Constrained Problems
por: Shustin, Boris, et al.
Publicado: (2021)

Provably Faster Gradient Descent via Long Steps
por: Grimmer, Benjamin
Publicado: (2023)

Muon is Not That Special: Random or Inverted Spectra Work Just as Well
por: Shumaylov, Zakhar, et al.
Publicado: (2026)

On Properties of Adjoint Systems for Evolutionary PDEs
por: Tran, Brian K., et al.
Publicado: (2024)

AutoBalance: An Automatic Balancing Framework for Training Physics-Informed Neural Networks
por: An, Kang, et al.
Publicado: (2025)

WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points
por: Li, Dongyue, et al.
Publicado: (2026)

Fast and Provable Tensor-Train Format Tensor Completion via Precondtioned Riemannian Gradient Descent
por: Bian, Fengmiao, et al.
Publicado: (2025)

Faster Computation of Entropic Optimal Transport via Stable Low Frequency Modes
por: Chhaibi, Reda, et al.
Publicado: (2025)

A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equations
por: Liu, Shu, et al.
Publicado: (2024)

Preconditioning transformations of adjoint systems for evolution equations
por: Tran, Brian K., et al.
Publicado: (2025)

Fast Unconstrained Optimization via Hessian Averaging and Adaptive Gradient Sampling Methods
por: O'Leary-Roseberry, Thomas, et al.
Publicado: (2024)

Dual Cone Gradient Descent for Training Physics-Informed Neural Networks
por: Hwang, Youngsik, et al.
Publicado: (2024)

Solving Dense Linear Systems Faster Than via Preconditioning
por: Dereziński, Michał, et al.
Publicado: (2023)

Shape Derivative-Informed Neural Operators with Application to Risk-Averse Shape Optimization
por: Gong, Xindi, et al.
Publicado: (2026)

Fine-grained Analysis and Faster Algorithms for Iteratively Solving Linear Systems
por: Dereziński, Michał, et al.
Publicado: (2024)

Faster Linear Systems and Matrix Norm Approximation via Multi-level Sketched Preconditioning
por: Dereziński, Michał, et al.
Publicado: (2024)

Extending Neural Operators: Robust Handling of Functions Beyond the Training Set
por: Quackenbush, Blaine, et al.
Publicado: (2026)

Dynamic Proximal Gradient Algorithms for Schatten-$p$ Quasi-Norm Regularized Problems
por: Shen, Weiping, et al.
Publicado: (2026)

Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noise
por: Fang, Yuchen, et al.
Publicado: (2026)

Natural Riemannian gradient for learning functional tensor networks
por: Klug, Nikolas, et al.
Publicado: (2026)

State-Dependent Lyapunov Method for Rank-1 Matrix Factorization
por: Moon, Jaehong
Publicado: (2026)

Primal-Dual Methods for Nonsmooth Nonconvex Optimization with Orthogonality Constraints
por: Zhu, Linglingzhi, et al.
Publicado: (2026)

A distributed semismooth Newton based augmented Lagrangian method for distributed optimization
por: Ma, Qihao, et al.
Publicado: (2026)

Solving the Offline and Online Min-Max Problem of Non-smooth Submodular-Concave Functions: A Zeroth-Order Approach
por: Farzin, Amir Ali, et al.
Publicado: (2026)

Last-Iterate Convergence of Randomized Kaczmarz and SGD with Greedy Step Size
por: Dereziński, Michał, et al.
Publicado: (2026)

OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality
por: Yuan, Ganzhao
Publicado: (2026)

Nonlinear model reduction for transport-dominated problems
por: Hesthaven, Jan S., et al.
Publicado: (2026)

Kernel-based potential mean-field games with unbiased random Fourier $U$-statistics
por: Nakano, Yumiharu
Publicado: (2026)

Adaptive Lipschitz-Free Conditional Gradient Methods for Stochastic Composite Nonconvex Optimization
por: Yuan, Ganzhao
Publicado: (2026)

Continuum-marginal optimal transport: a mesh-free kernel method
por: Nakano, Yumiharu
Publicado: (2026)

A Trust-Region Interior-Point Stochastic Sequential Quadratic Programming Method
por: Fang, Yuchen, et al.
Publicado: (2026)

Fundamental Bias in Inverting Random Sampling Matrices with Application to Sub-sampled Newton
por: Niu, Chengmei, et al.
Publicado: (2025)

Scalable Acceleration for Classification-Based Derivative-Free Optimization
por: Han, Tianyi, et al.
Publicado: (2023)

Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses
por: Dereich, Steffen, et al.
Publicado: (2024)

On the numerical reliability of nonsmooth autodiff: a MaxPool case study
por: Boustany, Ryan
Publicado: (2024)

Error Feedback Can Accurately Compress Preconditioners
por: Modoranu, Ionut-Vlad, et al.
Publicado: (2023)

Linear convergence of forward-backward accelerated algorithms without knowledge of the modulus of strong convexity
por: Li, Bowen, et al.
Publicado: (2023)

A Gauss-Newton Approach for Min-Max Optimization in Generative Adversarial Networks
por: Mishra, Neel, et al.
Publicado: (2024)

Subhomogeneous Deep Equilibrium Models
por: Sittoni, Pietro, et al.
Publicado: (2024)

Solving Elliptic Optimal Control Problems via Neural Networks and Optimality System
por: Dai, Yongcheng, et al.
Publicado: (2023)