Cover Image

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Longteng, Wu, Sen, Hou, Shuai, Qing, Zhengyu, Zheng, Zhuo, Ke, Danning, Lin, Qihong, Wang, Qiang, Shi, Shaohuai, Chu, Xiaowen
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
by: Zhang, Longteng, et al.
Published: (2023)

Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems
by: Lin, Wenxiang, et al.
Published: (2024)

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
by: Pan, Xinglin, et al.
Published: (2024)

Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
by: Pan, Xinglin, et al.
Published: (2025)

ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
by: Lin, Wenxiang, et al.
Published: (2026)

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
by: Tang, Zichen, et al.
Published: (2024)

On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning
by: Ahmad, Muhammad, et al.
Published: (2026)

Memory-Efficient Fine-Tuning via Low-Rank Activation Compression
by: Shi, Jiang-Xin, et al.
Published: (2025)

Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models
by: Zhao, Jialin, et al.
Published: (2025)

LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection
by: Zhao, Lanxin, et al.
Published: (2026)

SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
by: Lu, Xudong, et al.
Published: (2024)

NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models
by: Lin, Cheng, et al.
Published: (2024)

HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
by: Lin, Wenxiang, et al.
Published: (2025)

FAAR: Efficient Frequency-Aware Multi-Task Fine-Tuning via Automatic Rank Selection
by: Fontana, Maxime, et al.
Published: (2026)

Decentralized Low-Rank Fine-Tuning of Large Language Models
by: Ghiasvand, Sajjad, et al.
Published: (2025)

PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decoupling
by: Wang, Sijie, et al.
Published: (2025)

Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning
by: Han, Xiao, et al.
Published: (2025)

RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
by: Zhang, Yilang, et al.
Published: (2025)

MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning
by: Ren, Pengjie, et al.
Published: (2024)

Low-rank Attention Side-Tuning for Parameter-Efficient Fine-Tuning
by: Tang, Ningyuan, et al.
Published: (2024)

Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets
by: Bilican, Ahmet, et al.
Published: (2025)

Resource-Efficient Personal Large Language Models Fine-Tuning with Collaborative Edge Computing
by: Ye, Shengyuan, et al.
Published: (2024)

LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning
by: Du, Zhekai, et al.
Published: (2025)

RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair
by: Silva, André, et al.
Published: (2023)

Unlocking Multi-Task Electric Energy System Intelligence: Data Scaling Laws and Performance with Limited Fine-Tuning
by: Liu, Shaohuai, et al.
Published: (2025)

TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models
by: Cao, Yarui, et al.
Published: (2026)

AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models
by: Liu, Zeyu, et al.
Published: (2024)

AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption
by: Zhou, Yajie, et al.
Published: (2025)

FedImpro: Measuring and Improving Client Update in Federated Learning
by: Tang, Zhenheng, et al.
Published: (2024)

ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
by: Zhang, Yilang, et al.
Published: (2025)

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models
by: Yang, Yifan, et al.
Published: (2024)

Impact of Fine-Tuning Methods on Memorization in Large Language Models
by: Hou, Jie, et al.
Published: (2025)

R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
by: Zhang, Zhenyu, et al.
Published: (2025)

QR-LoRA: QR-Based Low-Rank Adaptation for Efficient Fine-Tuning of Large Language Models
by: Liang, Jessica, et al.
Published: (2025)

High-Rank Structured Modulation for Parameter-Efficient Fine-Tuning
by: Liu, Yongkang, et al.
Published: (2026)

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
by: Modoranu, Ionut-Vlad, et al.
Published: (2026)

QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation
by: Chen, Zhuo, et al.
Published: (2024)

Improving Anomalous Sound Detection via Low-Rank Adaptation Fine-Tuning of Pre-Trained Audio Models
by: Zheng, Xinhu, et al.
Published: (2024)

LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning
by: Shi, Yiming, et al.
Published: (2024)

DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
by: Zhang, Haojie
Published: (2025)