Cover Image

FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Modoranu, Ionut-Vlad, Safaryan, Mher, Schultheis, Erik, Ryabinin, Max, Chumachenko, Artem, Alistarh, Dan
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
by: Modoranu, Ionut-Vlad, et al.
Published: (2026)

LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics
by: Robert, Thomas, et al.
Published: (2024)

DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers
by: Modoranu, Ionut-Vlad, et al.
Published: (2026)

MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
by: Modoranu, Ionut-Vlad, et al.
Published: (2024)

The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information
by: Wu, Diyuan, et al.
Published: (2024)

Unified Scaling Laws for Compressed Representations
by: Panferov, Andrei, et al.
Published: (2025)

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
by: Jovanović, Andrej, et al.
Published: (2026)

Towards Robust Scaling Laws for Optimizers
by: Volkova, Alexandra, et al.
Published: (2026)

Error Feedback Can Accurately Compress Preconditioners
by: Modoranu, Ionut-Vlad, et al.
Published: (2023)

CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
by: Tabesh, Soroush, et al.
Published: (2025)

LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs
by: Schultheis, Erik, et al.
Published: (2025)

Mashup Learning: Faster Finetuning by Remixing Past Checkpoints
by: Vaina, Sofia Maria Lo Cicero, et al.
Published: (2026)

Optimizers Qualitatively Alter Solutions And We Should Leverage This
by: Pascanu, Razvan, et al.
Published: (2025)

Grid Games: The Power of Multiple Grids for Quantizing Large Language Models
by: Egiazarian, Vage, et al.
Published: (2026)

International security Jstor-WOS 2010-2024
by: Chumachenko, Artem
Published: (2025)

GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity
by: Maranjyan, Artavazd, et al.
Published: (2022)

Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation
by: Panferov, Andrei, et al.
Published: (2026)

On Biased Compression for Distributed Learning
by: Beznosikov, Aleksandr, et al.
Published: (2020)

Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models
by: Kurtic, Eldar, et al.
Published: (2024)

DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
by: Iacob, Alex, et al.
Published: (2025)

Extreme Compression of Large Language Models via Additive Quantization
by: Egiazarian, Vage, et al.
Published: (2024)

CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models
by: Xiao, Xiaojun, et al.
Published: (2024)

Statistically-Lossless Quantization of Large Language Models
by: Helcig, Michael, et al.
Published: (2026)

DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Models
by: Zhang, Jinbin, et al.
Published: (2025)

ELMO: Efficiency via Low-precision and Peak Memory Optimization in Large Output Spaces
by: Zhang, Jinbin, et al.
Published: (2025)

MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
by: Iacob, Alex, et al.
Published: (2025)

Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization
by: Ji, Yixin, et al.
Published: (2024)

Simple Opinion Dynamics for No-Regret Learning
by: Lazarsfeld, John, et al.
Published: (2023)

On the Low-Rank Parametrization of Reward Models for Controlled Language Generation
by: Troshin, Sergey, et al.
Published: (2024)

Data-Adaptive Low-Rank Sparse Subspace Clustering
by: Kopriva, Ivica
Published: (2025)

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models
by: Lu, Yuheng, et al.
Published: (2024)

LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusion
by: Zhou, Guanghao, et al.
Published: (2026)

DarwinLM: Evolutionary Structured Pruning of Large Language Models
by: Tang, Shengkun, et al.
Published: (2025)

Model Compression with Exact Budget Constraints via Riemannian Manifolds
by: Helcig, Michael, et al.
Published: (2026)

Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Models
by: Erden, Caner
Published: (2025)

Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inference
by: Samplawski, Colin, et al.
Published: (2025)

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
by: Shutova, Alina, et al.
Published: (2025)

Game Dynamics and Equilibrium Computation in the Population Protocol Model
by: Alistarh, Dan, et al.
Published: (2023)

Mixture-of-Subspaces in Low-Rank Adaptation
by: Wu, Taiqiang, et al.
Published: (2024)

Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements
by: Voronov, Anton, et al.
Published: (2024)