Cover Image

GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liao, Xutao, Li, Shaohui, Xu, Yuhui, Li, Zhi, Liu, Yu, He, You
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
by: Su, DiJia, et al.
Published: (2025)

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
by: Zhang, Zhenyu, et al.
Published: (2024)

TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition
by: Lin, Tianwei, et al.
Published: (2024)

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
by: Zhao, Jiawei, et al.
Published: (2024)

Natural GaLore: Accelerating GaLore for memory-efficient LLM Training and Fine-tuning
by: Das, Arijit
Published: (2024)

Less is More: Resource-Efficient Low-Rank Adaptation
by: Tian, Chunlin, et al.
Published: (2025)

LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
by: Zhang, Juzheng, et al.
Published: (2025)

Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations
by: Deng, Guanzhi, et al.
Published: (2025)

Flexora: Flexible Low Rank Adaptation for Large Language Models
by: Wei, Chenxing, et al.
Published: (2024)

LoRMA: Low-Rank Multiplicative Adaptation for LLMs
by: Bihany, Harsh, et al.
Published: (2025)

Toward Robust Multilingual Adaptation of LLMs for Low-Resource Languages
by: Li, Haolin, et al.
Published: (2025)

SBoRA: Low-Rank Adaptation with Regional Weight Updates
by: Po, Lai-Man, et al.
Published: (2024)

PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation
by: Benedek, Nadav, et al.
Published: (2024)

TsqLoRA: Towards Sensitivity and Quality Low-Rank Adaptation for Efficient Fine-Tuning
by: Chen, Yu, et al.
Published: (2025)

DenseLoRA: Dense Low-Rank Adaptation of Large Language Models
by: Mu, Lin, et al.
Published: (2025)

MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
by: Tang, Chuanyu, et al.
Published: (2024)

Ranking LLMs by compression
by: Guo, Peijia, et al.
Published: (2024)

Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat
by: Daynauth, Roland, et al.
Published: (2024)

SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skipping
by: Lu, Yu-Chen, et al.
Published: (2025)

The Expressive Power of Low-Rank Adaptation
by: Zeng, Yuchen, et al.
Published: (2023)

LoRA+: Efficient Low Rank Adaptation of Large Models
by: Hayou, Soufiane, et al.
Published: (2024)

Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation
by: Wang, Dianyun, et al.
Published: (2025)

SwitchLoRA: Switched Low-Rank Adaptation Can Learn Full-Rank Information
by: Zhou, Kaiye, et al.
Published: (2024)

Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs
by: Zhang, Yifei, et al.
Published: (2024)

GoRA: Gradient-driven Adaptive Low Rank Adaptation
by: He, Haonan, et al.
Published: (2025)

Debiasing LLMs by Masking Unfairness-Driving Attention Heads
by: Han, Tingxu, et al.
Published: (2025)

Batched Low-Rank Adaptation of Foundation Models
by: Wen, Yeming, et al.
Published: (2023)

Variational Low-Rank Adaptation Using IVON
by: Cong, Bai, et al.
Published: (2024)

MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning
by: Zhao, Jiancheng, et al.
Published: (2025)

LoRTA: Low Rank Tensor Adaptation of Large Language Models
by: Hounie, Ignacio, et al.
Published: (2024)

GLoRIA: Gated Low-Rank Interpretable Adaptation for Dialectal ASR
by: Mehralian, Pouya, et al.
Published: (2026)

Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE
by: Huang, Haiduo, et al.
Published: (2025)

LoR2C : Low-Rank Residual Connection Adaptation for Parameter-Efficient Fine-Tuning
by: Zhao, Jiancheng, et al.
Published: (2025)

AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation
by: Liu, Ziyun, et al.
Published: (2026)

TLoRA: Task-aware Low Rank Adaptation of Large Language Models
by: Lin, Weicheng, et al.
Published: (2026)

RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty
by: Zhang, Ziqian, et al.
Published: (2026)

LoTR: Low Tensor Rank Weight Adaptation
by: Bershatsky, Daniel, et al.
Published: (2024)

Low-Rank Adaptation for Multilingual Summarization: An Empirical Study
by: Whitehouse, Chenxi, et al.
Published: (2023)

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning
by: Liang, Yao, et al.
Published: (2024)

AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models
by: Liu, Zeyu, et al.
Published: (2024)