Cover Image

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Jaiswal, Ajay, Wang, Yifan, Yin, Lu, Liu, Shiwei, Chen, Runjin, Zhao, Jiawei, Grama, Ananth, Tian, Yuandong, Wang, Zhangyang
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
by: Zhang, Zhenyu, et al.
Published: (2024)

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
by: Zhao, Jiawei, et al.
Published: (2024)

Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs
by: Yin, Lu, et al.
Published: (2023)

GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
by: Su, DiJia, et al.
Published: (2025)

LLaGA: Large Language and Graph Assistant
by: Chen, Runjin, et al.
Published: (2024)

LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning
by: Perin, Gabriel J., et al.
Published: (2025)

Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
by: Fernandez-Lopez, Adriana, et al.
Published: (2024)

LLMs Can Get "Brain Rot": A Pilot Study on Twitter/X
by: Xing, Shuo, et al.
Published: (2025)

Mixture-of-Subspaces in Low-Rank Adaptation
by: Wu, Taiqiang, et al.
Published: (2024)

Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation
by: Song, Zhao, et al.
Published: (2023)

Regularizing Subspace Redundancy of Low-Rank Adaptation
by: Zhu, Yue, et al.
Published: (2025)

More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
by: Wang, Yifan, et al.
Published: (2025)

R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
by: Zhang, Zhenyu, et al.
Published: (2025)

Deconvolving Complex Neuronal Networks into Interpretable Task-Specific Connectomes
by: Wang, Yifan, et al.
Published: (2024)

InRank: Incremental Low-Rank Learning
by: Zhao, Jiawei, et al.
Published: (2023)

Adaptive Stochastic Gradient Descents on Manifolds with an Application on Weighted Low-Rank Approximation
by: Yang, Peiqi, et al.
Published: (2025)

DoRA: Weight-Decomposed Low-Rank Adaptation
by: Liu, Shih-Yang, et al.
Published: (2024)

Random Subspace Cubic-Regularization Methods, with Applications to Low-Rank Functions
by: Cartis, Coralia, et al.
Published: (2025)

Revisiting Weight Regularization for Low-Rank Continual Learning
by: Zheng, Yaoyue, et al.
Published: (2026)

Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining
by: Zhang, Haochen, et al.
Published: (2025)

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models
by: Lu, Yuheng, et al.
Published: (2024)

Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation
by: Wang, Dianyun, et al.
Published: (2025)

Data-Adaptive Low-Rank Sparse Subspace Clustering
by: Kopriva, Ivica
Published: (2025)

Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions
by: Ouyang, Yihao, et al.
Published: (2026)

RaSA: Rank-Sharing Low-Rank Adaptation
by: He, Zhiwei, et al.
Published: (2025)

SBoRA: Low-Rank Adaptation with Regional Weight Updates
by: Po, Lai-Man, et al.
Published: (2024)

Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching
by: Miao, Tianhao, et al.
Published: (2026)

Unbiased Gradient Low-Rank Projection
by: Pan, Rui, et al.
Published: (2025)

MAP: Revisiting Weight Decomposition for Low-Rank Adaptation
by: Si, Chongjie, et al.
Published: (2025)

An Overview of Low-Rank Structures in the Training and Adaptation of Large Models
by: Balzano, Laura, et al.
Published: (2025)

Inference-Time Code Selection via Symbolic Equivalence Partitioning
by: Cho, David, et al.
Published: (2026)

Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation
by: Steele, Brady
Published: (2026)

Low-Tubal-Rank Tensor Recovery via Factorized Gradient Descent
by: Liu, Zhiyu, et al.
Published: (2024)

ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
by: Li, Jiaxi, et al.
Published: (2026)

Sebra: Debiasing Through Self-Guided Bias Ranking
by: Kappiyath, Adarsh, et al.
Published: (2025)

LoRA-GA: Low-Rank Adaptation with Gradient Approximation
by: Wang, Shaowen, et al.
Published: (2024)

Orthogonal Low Rank Embedding Stabilization
by: Zielnicki, Kevin, et al.
Published: (2025)

Beyond Low-Rank: Low-Rank Sparse Prompting via Spiking Neural Network and Prompt Factorization
by: Zhao, Yumiao, et al.
Published: (2026)

Low-Rank Graphon Estimation: Theory and Applications to Graphon Games
by: Klopp, Olga, et al.
Published: (2025)

Reweighted Solutions for Weighted Low Rank Approximation
by: Woodruff, David P., et al.
Published: (2024)