Image de couverture de livre

Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Petrov, Egor, Evseev, Grigoriy, Antonov, Aleksey, Veprikov, Andrey, Bushkov, Nikolay, Moiseev, Stanislav, Beznosikov, Aleksandr
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Optimization and Control
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling
par: Parfenov, Valery, et autres
Publié: (2026)

Sign-SGD via Parameter-Free Optimization
par: Medyakov, Daniil, et autres
Publié: (2025)

Shuffling Heuristic in Variational Inequalities: Establishing New Convergence Guarantees
par: Medyakov, Daniil, et autres
Publié: (2025)

Phases of Muon: When Muon Eclipses SignSGD
par: Paquette, Elliot, et autres
Publié: (2026)

New Aspects of Black Box Conditional Gradient: Variance Reduction and One Point Feedback
par: Veprikov, Andrey, et autres
Publié: (2024)

Methods for Optimization Problems with Markovian Stochasticity and Non-Euclidean Geometry
par: Solodkin, Vladimir, et autres
Publié: (2024)

WeightLoRA: Keep Only Necessary Adapters
par: Veprikov, Andrey, et autres
Publié: (2025)

Markovian Compression: Looking to the Past Helps Accelerate the Future
par: Veprikov, Andrey, et autres
Publié: (2026)

Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
par: Kim, Jihwan, et autres
Publié: (2026)

StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
par: Yu, Dingzhi, et autres
Publié: (2026)

Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers
par: Riabinin, Artem, et autres
Publié: (2026)

Local SGD for Near-Quadratic Problems: Improving Convergence under Unconstrained Noise Conditions
par: Sadchikov, Andrey, et autres
Publié: (2024)

Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
par: Veprikov, Andrey, et autres
Publié: (2025)

When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds
par: Tao, Hongyi, et autres
Publié: (2026)

DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
par: Yudin, Nikolay, et autres
Publié: (2025)

LionMuon: Alternating Spectral and Sign Descent for Efficient Training
par: Bolatov, Arman, et autres
Publié: (2026)

Accelerated Zero-Order SGD Method for Solving the Black Box Optimization Problem under "Overparametrization" Condition
par: Lobanov, Aleksandr, et autres
Publié: (2023)

Convergence of Clipped-SGD for Convex $(L_0,L_1)$-Smooth Optimization with Heavy-Tailed Noise
par: Chezhegov, Savelii, et autres
Publié: (2025)

SignSGD with Federated Voting
par: Park, Chanho, et autres
Publié: (2024)

Extragradient Sliding for Composite Non-Monotone Variational Inequalities
par: Emelyanov, Roman, et autres
Publié: (2024)

Unified Theory of Adaptive Variance Reduction
par: Shestakov, Aleksandr, et autres
Publié: (2025)

Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
par: Feoktistov, Dmitrii, et autres
Publié: (2026)

Random-reshuffled SARAH does not need a full gradient computations
par: Beznosikov, Aleksandr, et autres
Publié: (2021)

Ito Diffusion Approximation of Universal Ito Chains for Sampling, Optimization and Boosting
par: Ustimenko, Aleksei, et autres
Publié: (2023)

Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update
par: Almansoori, Abdulla Jasem, et autres
Publié: (2025)

Effective Method with Compression for Distributed and Federated Cocoercive Variational Inequalities
par: Medyakov, Daniil, et autres
Publié: (2024)

Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC
par: Almansoori, Abdulla Jasem, et autres
Publié: (2026)

Accelerated Methods with Compressed Communications for Distributed Optimization Problems under Data Similarity
par: Bylinkin, Dmitry, et autres
Publié: (2024)

Dimension-adapted Momentum Outscales SGD
par: Ferbach, Damien, et autres
Publié: (2025)

Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning
par: Bylinkin, Dmitry, et autres
Publié: (2024)

Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features
par: Beznosikov, Aleksandr, et autres
Publié: (2023)

On the Convergence of DP-SGD with Adaptive Clipping
par: Shulgin, Egor, et autres
Publié: (2024)

First Order Methods with Markovian Noise: from Acceleration to Variational Inequalities
par: Beznosikov, Aleksandr, et autres
Publié: (2023)

Avoiding Bias in Clipped SGD for Overparameterized Models under Generalized Smoothness
par: Lobanov, Aleksandr, et autres
Publié: (2026)

Methods for Solving Variational Inequalities with Markovian Stochasticity
par: Solodkin, Vladimir, et autres
Publié: (2024)

Incorporating Preconditioning into Accelerated Approaches: Theoretical Guarantees and Practical Improvement
par: Trifonov, Stepan, et autres
Publié: (2025)

Accelerated Methods with Compression for Horizontal and Vertical Federated Learning
par: Stanko, Sergey, et autres
Publié: (2024)

Optimal Analysis of Method with Batching for Monotone Stochastic Finite-Sum Variational Inequalities
par: Pichugin, Alexander, et autres
Publié: (2024)

Power of Generalized Smoothness in Stochastic Convex Optimization: First- and Zero-Order Algorithms
par: Lobanov, Aleksandr, et autres
Publié: (2025)

Hierarchical Mixture-of-Experts with Two-Stage Optimization
par: Molodtsov, Gleb, et autres
Publié: (2026)