Cover Image

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Sun, Mengyang, Wang, Yihao, Feng, Tao, Zhang, Dan, Zhu, Yifan, Tang, Jie
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
by: Sun, Mengyang, et al.
Published: (2026)

Parameter-Efficient Fine-Tuning for Foundation Models
by: Zhang, Dan, et al.
Published: (2025)

SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models
by: Tang, Anke, et al.
Published: (2024)

Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning
by: Cong, Peizhuang, et al.
Published: (2025)

HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
by: Wei, Jia, et al.
Published: (2026)

MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
by: Tang, Chuanyu, et al.
Published: (2024)

PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning
by: Feng, Yu, et al.
Published: (2025)

Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
by: Guo, Yongxin, et al.
Published: (2024)

PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model
by: Liu, Yilun, et al.
Published: (2024)

Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
by: Zhao, Ziyu, et al.
Published: (2025)

MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
by: Raje, Arian, et al.
Published: (2026)

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models
by: Yang, Yifan, et al.
Published: (2024)

Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training
by: Zhang, Chengqian, et al.
Published: (2026)

SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning
by: Xie, Zhen-Hao, et al.
Published: (2026)

Memory-Efficient Fine-Tuning via Low-Rank Activation Compression
by: Shi, Jiang-Xin, et al.
Published: (2025)

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
by: Zhang, Longteng, et al.
Published: (2026)

T-REX: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning
by: Zhang, Rongyu, et al.
Published: (2024)

WaveMoE: A Wavelet-Enhanced Mixture-of-Experts Foundation Model for Time Series Forecasting
by: Wu, Shunyu, et al.
Published: (2026)

Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts
by: Zhang, Buze, et al.
Published: (2026)

Polynomial Expansion Rank Adaptation: Enhancing Low-Rank Fine-Tuning with High-Order Interactions
by: Zhang, Wenhao, et al.
Published: (2026)

H2Tune: Federated Foundation Model Fine-Tuning with Hybrid Heterogeneity
by: Guo, Wei, et al.
Published: (2025)

Efficient Handwriting-Based Alzheimer,s Disease Diagnosis Using a Low-Rank Mixture of Experts Deep Learning Framework
by: Wang, Wu, et al.
Published: (2026)

MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
by: Yang, Cheng, et al.
Published: (2024)

L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
by: Yang, Minghao, et al.
Published: (2026)

Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
by: Liu, Yilun, et al.
Published: (2025)

RevFFN: Memory-Efficient Full-Parameter Fine-Tuning of Mixture-of-Experts LLMs with Reversible Blocks
by: Liu, Ningyuan, et al.
Published: (2025)

A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning
by: Bian, Jieming, et al.
Published: (2025)

Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptation
by: Jain, Abhinav, et al.
Published: (2024)

Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
by: Wang, Chenxu, et al.
Published: (2025)

Mixture-of-Subspaces in Low-Rank Adaptation
by: Wu, Taiqiang, et al.
Published: (2024)

Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
by: Tang, Yuxuan, et al.
Published: (2025)

FedPFT: Federated Proxy Fine-Tuning of Foundation Models
by: Peng, Zhaopeng, et al.
Published: (2024)

Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning
by: Raje, Arian, et al.
Published: (2025)

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
by: Modoranu, Ionut-Vlad, et al.
Published: (2026)

Domain-Aware Fine-Tuning of Foundation Models
by: Kaplan, Ugur Ali, et al.
Published: (2024)

TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
by: Kunwar, Pradip, et al.
Published: (2025)

Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models
by: Liang, Jingcong, et al.
Published: (2025)

Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models
by: Tang, Yuanbo, et al.
Published: (2025)

L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts
by: Ji, Shihao, et al.
Published: (2025)

Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts
by: Shi, Xiaoming, et al.
Published: (2024)