Cover Image

Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Miao, Tianhao, Bao, Zhongyuan, Zhang, Lejun
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Gradient Weight-normalized Low-rank Projection for Efficient LLM Training
by: Huang, Jia-Hong, et al.
Published: (2024)

GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
by: Su, DiJia, et al.
Published: (2025)

Unbiased Gradient Low-Rank Projection
by: Pan, Rui, et al.
Published: (2025)

Mixture-of-Subspaces in Low-Rank Adaptation
by: Wu, Taiqiang, et al.
Published: (2024)

Low-Rank Compression of Pretrained Models via Randomized Subspace Iteration
by: Pourkamali-Anaraki, Farhad
Published: (2026)

FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models
by: Modoranu, Ionut-Vlad, et al.
Published: (2025)

Memory-Efficient LLM Training with Online Subspace Descent
by: Liang, Kaizhao, et al.
Published: (2024)

Catching a Moving Subspace: Low-Rank Bandits Beyond Stationarity
by: Khosravi, Hamed, et al.
Published: (2026)

CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models
by: Xiao, Xiaojun, et al.
Published: (2024)

GoRA: Gradient-driven Adaptive Low Rank Adaptation
by: He, Haonan, et al.
Published: (2025)

Robust and Efficient Zeroth-Order LLM Fine-Tuning via Adaptive Bayesian Subspace Optimizer
by: Feng, Jian, et al.
Published: (2026)

Projecting Out the Malice: A Global Subspace Approach to LLM Detoxification
by: Duan, Zenghao, et al.
Published: (2026)

CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
by: Liu, Ziyue, et al.
Published: (2025)

Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation
by: Qi, Jun, et al.
Published: (2025)

Low Rank Gradients and Where to Find Them
by: Sonthalia, Rishi, et al.
Published: (2025)

CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization
by: Yang, Zi, et al.
Published: (2024)

Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
by: Wang, Chenxu, et al.
Published: (2025)

SwitchLoRA: Switched Low-Rank Adaptation Can Learn Full-Rank Information
by: Zhou, Kaiye, et al.
Published: (2024)

Flora: Low-Rank Adapters Are Secretly Gradient Compressors
by: Hao, Yongchang, et al.
Published: (2024)

Matrix Low-Rank Approximation For Policy Gradient Methods
by: Rozada, Sergio, et al.
Published: (2024)

Adaptive Budget Allocation for Orthogonal-Subspace Adapter Tuning in LLMs Continual Learning
by: Wan, Zhiyi, et al.
Published: (2025)

Palu: Compressing KV-Cache with Low-Rank Projection
by: Chang, Chi-Chih, et al.
Published: (2024)

EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training
by: Yi, Qingao, et al.
Published: (2025)

ROSA: Random Subspace Adaptation for Efficient Fine-Tuning
by: Hameed, Marawan Gamal Abdel, et al.
Published: (2024)

CALR: Corrective Adaptive Low-Rank Decomposition for Efficient Large Language Model Layer Compression
by: Kautsar, Muchammad Daniyal, et al.
Published: (2025)

Graph Ranking Contrastive Learning: A Extremely Simple yet Efficient Method
by: Hu, Yulan, et al.
Published: (2023)

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
by: Zhao, Jiawei, et al.
Published: (2024)

Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation
by: Wang, Dianyun, et al.
Published: (2025)

NeuraLSP: An Efficient and Rigorous Neural Left Singular Subspace Preconditioner for Conjugate Gradient Methods
by: Benanti, Alexander, et al.
Published: (2026)

Harnessing Orthogonality to Train Low-Rank Neural Networks
by: Coquelin, Daniel, et al.
Published: (2024)

Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
by: Jin, Gaojie, et al.
Published: (2026)

MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMs
by: Li, Guangyan, et al.
Published: (2025)

Hierarchical Sparse Plus Low Rank Compression of LLM
by: Kumar, Pawan, et al.
Published: (2025)

Probe-Free Low-Rank Activation Intervention
by: Jiang, Chonghe, et al.
Published: (2025)

Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension
by: Gong, Wenbo, et al.
Published: (2025)

Gradient-Free Training of Spiking Neural Networks via Low-Rank Evolution Strategies
by: Patankar, Dhruv, et al.
Published: (2026)

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
by: Lv, Xingtai, et al.
Published: (2024)

Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptation
by: Zhang, Xiyang, et al.
Published: (2026)

Emergent Low-Rank Training Dynamics in MLPs with Smooth Activations
by: Xu, Alec S., et al.
Published: (2026)

Gradient Multi-Normalization for Stateless and Scalable LLM Training
by: Scetbon, Meyer, et al.
Published: (2025)