Cover Image

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhao, Jiawei, Zhang, Zhenyu, Chen, Beidi, Wang, Zhangyang, Anandkumar, Anima, Tian, Yuandong
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
by: Zhang, Zhenyu, et al.
Published: (2024)

GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
by: Su, DiJia, et al.
Published: (2025)

Natural GaLore: Accelerating GaLore for memory-efficient LLM Training and Fine-tuning
by: Das, Arijit
Published: (2024)

InRank: Incremental Low-Rank Learning
by: Zhao, Jiawei, et al.
Published: (2023)

GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection
by: Liao, Xutao, et al.
Published: (2024)

Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Training
by: Luo, Cheng, et al.
Published: (2024)

TensorGRaD: Tensor Gradient Robust Decomposition for Memory-Efficient Neural Operator Training
by: Loeschcke, Sebastian, et al.
Published: (2025)

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
by: Jaiswal, Ajay, et al.
Published: (2024)

R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
by: Zhang, Zhenyu, et al.
Published: (2025)

LoCoCo: Dropping In Convolutions for Long Context Compression
by: Cai, Ruisi, et al.
Published: (2024)

HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
by: Luo, Cheng, et al.
Published: (2025)

Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients
by: Wang, Yezhen, et al.
Published: (2025)

Diffusion State-Guided Projected Gradient for Inverse Problems
by: Zirvi, Rayhan, et al.
Published: (2024)

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
by: Dong, Harry, et al.
Published: (2024)

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
by: Tian, Yuandong, et al.
Published: (2023)

MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models
by: Zhang, Junyang, et al.
Published: (2025)

Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching
by: Miao, Tianhao, et al.
Published: (2026)

Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs
by: George, Robert Joseph, et al.
Published: (2022)

Calibrated Uncertainty Quantification for Operator Learning via Conformal Prediction
by: Ma, Ziqi, et al.
Published: (2024)

Fourier Neural Operators Explained: A Practical Perspective
by: Duruisseaux, Valentin, et al.
Published: (2025)

Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients
by: Muhamed, Aashiq, et al.
Published: (2024)

Gradient Weight-normalized Low-rank Projection for Efficient LLM Training
by: Huang, Jia-Hong, et al.
Published: (2024)

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
by: Sun, Hanshi, et al.
Published: (2024)

Fast Training of Diffusion Models with Masked Transformers
by: Zheng, Hongkai, et al.
Published: (2023)

FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection
by: He, Yutong, et al.
Published: (2026)

MagicPIG: LSH Sampling for Efficient LLM Generation
by: Chen, Zhuoming, et al.
Published: (2024)

Unbiased Gradient Low-Rank Projection
by: Pan, Rui, et al.
Published: (2025)

Deep Think with Confidence
by: Fu, Yichao, et al.
Published: (2025)

Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
by: Zheng, Haizhong, et al.
Published: (2025)

Mechanistic Interpretability with Sparse Autoencoder Neural Operators
by: Tolooshams, Bahareh, et al.
Published: (2025)

AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning
by: Refael, Yehonathan, et al.
Published: (2024)

Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning
by: Wang, Peihao, et al.
Published: (2025)

Geometric Operator Learning with Optimal Transport
by: Li, Xinyi, et al.
Published: (2025)

Solving Poisson Equations using Neural Walk-on-Spheres
by: Nam, Hong Chul, et al.
Published: (2024)

Lean Copilot: Large Language Models as Copilots for Theorem Proving in Lean
by: Song, Peiyang, et al.
Published: (2024)

Memory-Efficient Differentially Private Training with Gradient Random Projection
by: Mulrooney, Alex, et al.
Published: (2025)

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
by: Zhang, Zhenyu, et al.
Published: (2024)

S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity
by: Yang, Xinyu, et al.
Published: (2024)

LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM
by: Refael, Yehonathan, et al.
Published: (2025)

AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections
by: Yu, Xin, et al.
Published: (2025)