Cover Image

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhao, Ziyu, Shen, Tao, Zhu, Didi, Li, Zexi, Su, Jing, Wang, Xuwu, Kuang, Kun, Wu, Fei
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
by: Zhao, Ziyu, et al.
Published: (2025)

LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks
by: Prabhakar, Akshara, et al.
Published: (2024)

ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs
by: Shah, Viraj, et al.
Published: (2023)

Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing
by: Cheng, Ming, et al.
Published: (2025)

K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs
by: Ouyang, Ziheng, et al.
Published: (2025)

LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging
by: Lee, Seungeon, et al.
Published: (2025)

LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
by: Li, Wenbing, et al.
Published: (2025)

The Appeal and Reality of Recycling LoRAs with Adaptive Merging
by: Liu, Haokun, et al.
Published: (2026)

Tensorized Clustered LoRA Merging for Multi-Task Interference
by: Su, Zhan, et al.
Published: (2025)

Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning
by: Zhao, Ziyu, et al.
Published: (2024)

LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation
by: Shenaj, Donald, et al.
Published: (2024)

Towards Modular LLMs by Building and Reusing a Library of LoRAs
by: Ostapenko, Oleksiy, et al.
Published: (2024)

Video2LoRA: Unified Semantic-Controlled Video Generation via Per-Reference-Video LoRA
by: Wu, Zexi, et al.
Published: (2026)

SCALE-LoRA: Auditing Post-Retrieval LoRA Composition with Residual Merging and View Reliability
by: Zhou, Shuaipeng, et al.
Published: (2026)

Rank-1 LoRAs Encode Interpretable Reasoning Signals
by: Ward, Jake, et al.
Published: (2025)

DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
by: Deng, Guanzhi, et al.
Published: (2026)

CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models
by: Chen, Guanduo, et al.
Published: (2025)

S-LoRA: Serving Thousands of Concurrent LoRA Adapters
by: Sheng, Ying, et al.
Published: (2023)

LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild
by: Zhao, Ziyu, et al.
Published: (2024)

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree
by: Qian, Yu-Yang, et al.
Published: (2025)

Run LoRA Run: Faster and Lighter LoRA Implementations
by: Cherniuk, Daria, et al.
Published: (2023)

Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
by: Xu, Yixing, et al.
Published: (2025)

HRP: High-Rank Preheating for Superior LoRA Initialization
by: Chen, Yuzhu, et al.
Published: (2025)

SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
by: Luo, Minrui, et al.
Published: (2025)

Improving Recursive Transformers with Mixture of LoRAs
by: Nouriborji, Mohammadmahdi, et al.
Published: (2025)

LoBAM: LoRA-Based Backdoor Attack on Model Merging
by: Yin, Ming, et al.
Published: (2024)

LoRA as Oracle
by: Arazzi, Marco, et al.
Published: (2026)

Vision as LoRA
by: Wang, Han, et al.
Published: (2025)

LEGO: LoRA-Enabled Generator-Oriented Framework for Synthetic Image Detection
by: Xiao, Yutong, et al.
Published: (2026)

FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA
by: Zhang, Haoran, et al.
Published: (2026)

LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference
by: Rajabzadeh, Hossein, et al.
Published: (2026)

TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Experts
by: Jeon, Jeimin, et al.
Published: (2026)

PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization
by: Meng, Xiangdi, et al.
Published: (2024)

LoRAGuard: An Effective Black-box Watermarking Approach for LoRAs
by: Lv, Peizhuo, et al.
Published: (2025)

FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation
by: Sathyavageeswaran, Ramakrishnan
Published: (2026)

LangVision-LoRA-NAS: Neural Architecture Search for Variable LoRA Rank in Vision Language Models
by: Chitty-Venkata, Krishna Teja, et al.
Published: (2025)

qa-FLoRA: Data-free query-adaptive Fusion of LoRAs for LLMs
by: Shukla, Shreya, et al.
Published: (2025)

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
by: Bałazy, Klaudia, et al.
Published: (2024)

ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads
by: Zuo, Jingwei, et al.
Published: (2026)

LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement
by: Bian, Jieming, et al.
Published: (2024)