Copertina

Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Veprikov, Andrey, Bolatov, Arman, Bogdanov, Aleksandr, Horváth, Samuel, Beznosikov, Aleksandr, Takáč, Martin, Hanzely, Slavomir
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Optimization and Control
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers
di: Riabinin, Artem, et al.
Pubblicazione: (2026)

Loss-Transformation Invariance in the Damped Newton Method
di: Shestakov, Alexander, et al.
Pubblicazione: (2025)

Simple Stepsize for Quasi-Newton Methods with Global Convergence Guarantees
di: Agafonov, Artem, et al.
Pubblicazione: (2025)

New Aspects of Black Box Conditional Gradient: Variance Reduction and One Point Feedback
di: Veprikov, Andrey, et al.
Pubblicazione: (2024)

Newton Method Revisited: Global Convergence Rates up to $\mathcal {O}\left(k^{-3} \right)$ for Stepsize Schedules and Linesearch Procedures
di: Hanzely, Slavomír, et al.
Pubblicazione: (2024)

Adaptive Regularized Newton Method with Inexact Hessian
di: Shestakov, Aleksandr, et al.
Pubblicazione: (2025)

Sketch-and-Project Meets Newton Method: Global $\mathcal O(k^{-2})$ Convergence with Low-Rank Updates
di: Hanzely, Slavomír
Pubblicazione: (2023)

Methods for Optimization Problems with Markovian Stochasticity and Non-Euclidean Geometry
di: Solodkin, Vladimir, et al.
Pubblicazione: (2024)

Polyak Stepsize: Estimating Optimal Functional Values Without Parameters or Prior Knowledge
di: Abdukhakimov, Farshed, et al.
Pubblicazione: (2025)

Random-reshuffled SARAH does not need a full gradient computations
di: Beznosikov, Aleksandr, et al.
Pubblicazione: (2021)

LionMuon: Alternating Spectral and Sign Descent for Efficient Training
di: Bolatov, Arman, et al.
Pubblicazione: (2026)

Unified Theory of Adaptive Variance Reduction
di: Shestakov, Aleksandr, et al.
Pubblicazione: (2025)

Stochastic Gradient Methods with Preconditioned Updates
di: Sadiev, Abdurakhmon, et al.
Pubblicazione: (2022)

Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
di: Chezhegov, Savelii, et al.
Pubblicazione: (2024)

Markovian Compression: Looking to the Past Helps Accelerate the Future
di: Veprikov, Andrey, et al.
Pubblicazione: (2026)

WeightLoRA: Keep Only Necessary Adapters
di: Veprikov, Andrey, et al.
Pubblicazione: (2025)

Linear Convergence Rate in Convex Setup is Possible! Gradient Descent Method Variants under $(L_0,L_1)$-Smoothness
di: Lobanov, Aleksandr, et al.
Pubblicazione: (2024)

Incorporating Preconditioning into Accelerated Approaches: Theoretical Guarantees and Practical Improvement
di: Trifonov, Stepan, et al.
Pubblicazione: (2025)

Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling
di: Parfenov, Valery, et al.
Pubblicazione: (2026)

Convergence of Clipped-SGD for Convex $(L_0,L_1)$-Smooth Optimization with Heavy-Tailed Noise
di: Chezhegov, Savelii, et al.
Pubblicazione: (2025)

Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities
di: Beznosikov, Aleksandr, et al.
Pubblicazione: (2023)

$ψ$DAG: Projected Stochastic Approximation Iteration for DAG Structure Learning
di: Ziu, Klea, et al.
Pubblicazione: (2024)

Effective Method with Compression for Distributed and Federated Cocoercive Variational Inequalities
di: Medyakov, Daniil, et al.
Pubblicazione: (2024)

Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
di: Petrov, Egor, et al.
Pubblicazione: (2025)

Stochastic Frank-Wolfe: Unified Analysis and Zoo of Special Cases
di: Nazykov, Ruslan, et al.
Pubblicazione: (2024)

On Biased Compression for Distributed Learning
di: Beznosikov, Aleksandr, et al.
Pubblicazione: (2020)

Extragradient Sliding for Composite Non-Monotone Variational Inequalities
di: Emelyanov, Roman, et al.
Pubblicazione: (2024)

Local SGD for Near-Quadratic Problems: Improving Convergence under Unconstrained Noise Conditions
di: Sadchikov, Andrey, et al.
Pubblicazione: (2024)

Local Methods with Adaptivity via Scaling
di: Chezhegov, Savelii, et al.
Pubblicazione: (2024)

Accelerated Methods with Compressed Communications for Distributed Optimization Problems under Data Similarity
di: Bylinkin, Dmitry, et al.
Pubblicazione: (2024)

Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features
di: Beznosikov, Aleksandr, et al.
Pubblicazione: (2023)

Methods for Solving Variational Inequalities with Markovian Stochasticity
di: Solodkin, Vladimir, et al.
Pubblicazione: (2024)

Accelerated Methods with Compression for Horizontal and Vertical Federated Learning
di: Stanko, Sergey, et al.
Pubblicazione: (2024)

Ito Diffusion Approximation of Universal Ito Chains for Sampling, Optimization and Boosting
di: Ustimenko, Aleksei, et al.
Pubblicazione: (2023)

Optimal Analysis of Method with Batching for Monotone Stochastic Finite-Sum Variational Inequalities
di: Pichugin, Alexander, et al.
Pubblicazione: (2024)

DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
di: Yudin, Nikolay, et al.
Pubblicazione: (2025)

Bregman Proximal Method for Efficient Communications under Similarity
di: Beznosikov, Aleksandr, et al.
Pubblicazione: (2023)

Method with Batching for Stochastic Finite-Sum Variational Inequalities in Non-Euclidean Setting
di: Pichugin, Alexander, et al.
Pubblicazione: (2024)

Accelerated Stochastic Gradient Method with Applications to Consensus Problem in Markov-Varying Networks
di: Solodkin, Vladimir, et al.
Pubblicazione: (2024)

Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning
di: Bylinkin, Dmitry, et al.
Pubblicazione: (2024)