Cover Image

Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Almansoori, Abdulla Jasem, Ivanova, Maria, Veprikov, Andrey, Beznosikov, Aleksandr, Horváth, Samuel, Takáč, Martin
Format:	Preprint
Published:	2026
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update
by: Almansoori, Abdulla Jasem, et al.
Published: (2025)

Collaborative and Efficient Personalization with Mixtures of Adaptors
by: Almansoori, Abdulla Jasem, et al.
Published: (2024)

PaDPaF: Partial Disentanglement with Partially-Federated GANs
by: Almansoori, Abdulla Jasem, et al.
Published: (2022)

WeightLoRA: Keep Only Necessary Adapters
by: Veprikov, Andrey, et al.
Published: (2025)

Stochastic Gradient Methods with Preconditioned Updates
by: Sadiev, Abdurakhmon, et al.
Published: (2022)

Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
by: Veprikov, Andrey, et al.
Published: (2025)

LionMuon: Alternating Spectral and Sign Descent for Efficient Training
by: Bolatov, Arman, et al.
Published: (2026)

Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers
by: Riabinin, Artem, et al.
Published: (2026)

FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training
by: Zmushko, Philip, et al.
Published: (2024)

New Aspects of Black Box Conditional Gradient: Variance Reduction and One Point Feedback
by: Veprikov, Andrey, et al.
Published: (2024)

Convergence of Clipped-SGD for Convex $(L_0,L_1)$-Smooth Optimization with Heavy-Tailed Noise
by: Chezhegov, Savelii, et al.
Published: (2025)

Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
by: Petrov, Egor, et al.
Published: (2025)

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA
by: Rahulamathavan, Yogachandran, et al.
Published: (2026)

Random-reshuffled SARAH does not need a full gradient computations
by: Beznosikov, Aleksandr, et al.
Published: (2021)

Methods for Optimization Problems with Markovian Stochasticity and Non-Euclidean Geometry
by: Solodkin, Vladimir, et al.
Published: (2024)

Run LoRA Run: Faster and Lighter LoRA Implementations
by: Cherniuk, Daria, et al.
Published: (2023)

Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving
by: Wang, Xinyu, et al.
Published: (2025)

Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
by: Feoktistov, Dmitrii, et al.
Published: (2026)

Markovian Compression: Looking to the Past Helps Accelerate the Future
by: Veprikov, Andrey, et al.
Published: (2026)

T-LoRA: Single Image Diffusion Model Customization Without Overfitting
by: Soboleva, Vera, et al.
Published: (2025)

Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities
by: Beznosikov, Aleksandr, et al.
Published: (2023)

FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
by: Lee, Seanie, et al.
Published: (2025)

Local SGD for Near-Quadratic Problems: Improving Convergence under Unconstrained Noise Conditions
by: Sadchikov, Andrey, et al.
Published: (2024)

SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
by: Luo, Minrui, et al.
Published: (2025)

Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysis
by: Luo, Ruichen, et al.
Published: (2025)

Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling
by: Parfenov, Valery, et al.
Published: (2026)

Shared LoRA Subspaces for almost Strict Continual Learning
by: Kaushik, Prakhar, et al.
Published: (2026)

On Biased Compression for Distributed Learning
by: Beznosikov, Aleksandr, et al.
Published: (2020)

S-LoRA: Serving Thousands of Concurrent LoRA Adapters
by: Sheng, Ying, et al.
Published: (2023)

Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging
by: Zhang, Haobo, et al.
Published: (2025)

IterIS: Iterative Inference-Solving Alignment for LoRA Merging
by: Chen, Hongxu, et al.
Published: (2024)

Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
by: Xu, Yixing, et al.
Published: (2025)

Aligning Distributionally Robust Optimization with Practical Deep Learning Needs
by: Feoktistov, Dmitrii, et al.
Published: (2025)

Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
by: Chezhegov, Savelii, et al.
Published: (2024)

Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning
by: Feng, Zhi-Quan, et al.
Published: (2026)

Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment
by: Peng, Hongyi, et al.
Published: (2026)

LoRA as Oracle
by: Arazzi, Marco, et al.
Published: (2026)

Vision as LoRA
by: Wang, Han, et al.
Published: (2025)

DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
by: Yudin, Nikolay, et al.
Published: (2025)

FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA
by: Zhang, Haoran, et al.
Published: (2026)