Imagen de Portada

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Jaiswal, Ajay, Wang, Yifan, Yin, Lu, Liu, Shiwei, Chen, Runjin, Zhao, Jiawei, Grama, Ananth, Tian, Yuandong, Wang, Zhangyang
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
por: Zhang, Zhenyu, et al.
Publicado: (2024)

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
por: Zhao, Jiawei, et al.
Publicado: (2024)

Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs
por: Yin, Lu, et al.
Publicado: (2023)

GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
por: Su, DiJia, et al.
Publicado: (2025)

LLaGA: Large Language and Graph Assistant
por: Chen, Runjin, et al.
Publicado: (2024)

LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning
por: Perin, Gabriel J., et al.
Publicado: (2025)

Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
por: Fernandez-Lopez, Adriana, et al.
Publicado: (2024)

LLMs Can Get "Brain Rot": A Pilot Study on Twitter/X
por: Xing, Shuo, et al.
Publicado: (2025)

Mixture-of-Subspaces in Low-Rank Adaptation
por: Wu, Taiqiang, et al.
Publicado: (2024)

Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation
por: Song, Zhao, et al.
Publicado: (2023)

Regularizing Subspace Redundancy of Low-Rank Adaptation
por: Zhu, Yue, et al.
Publicado: (2025)

More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
por: Wang, Yifan, et al.
Publicado: (2025)

R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
por: Zhang, Zhenyu, et al.
Publicado: (2025)

Deconvolving Complex Neuronal Networks into Interpretable Task-Specific Connectomes
por: Wang, Yifan, et al.
Publicado: (2024)

InRank: Incremental Low-Rank Learning
por: Zhao, Jiawei, et al.
Publicado: (2023)

Adaptive Stochastic Gradient Descents on Manifolds with an Application on Weighted Low-Rank Approximation
por: Yang, Peiqi, et al.
Publicado: (2025)

DoRA: Weight-Decomposed Low-Rank Adaptation
por: Liu, Shih-Yang, et al.
Publicado: (2024)

Random Subspace Cubic-Regularization Methods, with Applications to Low-Rank Functions
por: Cartis, Coralia, et al.
Publicado: (2025)

Revisiting Weight Regularization for Low-Rank Continual Learning
por: Zheng, Yaoyue, et al.
Publicado: (2026)

Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining
por: Zhang, Haochen, et al.
Publicado: (2025)

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models
por: Lu, Yuheng, et al.
Publicado: (2024)

Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation
por: Wang, Dianyun, et al.
Publicado: (2025)

Data-Adaptive Low-Rank Sparse Subspace Clustering
por: Kopriva, Ivica
Publicado: (2025)

Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions
por: Ouyang, Yihao, et al.
Publicado: (2026)

RaSA: Rank-Sharing Low-Rank Adaptation
por: He, Zhiwei, et al.
Publicado: (2025)

SBoRA: Low-Rank Adaptation with Regional Weight Updates
por: Po, Lai-Man, et al.
Publicado: (2024)

Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching
por: Miao, Tianhao, et al.
Publicado: (2026)

Unbiased Gradient Low-Rank Projection
por: Pan, Rui, et al.
Publicado: (2025)

MAP: Revisiting Weight Decomposition for Low-Rank Adaptation
por: Si, Chongjie, et al.
Publicado: (2025)

An Overview of Low-Rank Structures in the Training and Adaptation of Large Models
por: Balzano, Laura, et al.
Publicado: (2025)

Inference-Time Code Selection via Symbolic Equivalence Partitioning
por: Cho, David, et al.
Publicado: (2026)

Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation
por: Steele, Brady
Publicado: (2026)

Low-Tubal-Rank Tensor Recovery via Factorized Gradient Descent
por: Liu, Zhiyu, et al.
Publicado: (2024)

ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
por: Li, Jiaxi, et al.
Publicado: (2026)

Sebra: Debiasing Through Self-Guided Bias Ranking
por: Kappiyath, Adarsh, et al.
Publicado: (2025)

LoRA-GA: Low-Rank Adaptation with Gradient Approximation
por: Wang, Shaowen, et al.
Publicado: (2024)

Orthogonal Low Rank Embedding Stabilization
por: Zielnicki, Kevin, et al.
Publicado: (2025)

Beyond Low-Rank: Low-Rank Sparse Prompting via Spiking Neural Network and Prompt Factorization
por: Zhao, Yumiao, et al.
Publicado: (2026)

Low-Rank Graphon Estimation: Theory and Applications to Graphon Games
por: Klopp, Olga, et al.
Publicado: (2025)

Reweighted Solutions for Weighted Low Rank Approximation
por: Woodruff, David P., et al.
Publicado: (2024)