Image de couverture de livre

Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Cong, Peizhuang, Liu, Wenpu, Yu, Wenhan, Zhao, Haochen, Yang, Tong
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
par: Sun, Mengyang, et autres
Publié: (2025)

TuckA: Hierarchical Compact Tensor Experts for Efficient Fine-Tuning
par: Lei, Qifeng, et autres
Publié: (2025)

Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing
par: Cong, Peizhuang, et autres
Publié: (2024)

FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering
par: Ping, Siqi, et autres
Publié: (2024)

PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model
par: Liu, Yilun, et autres
Publié: (2024)

Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
par: Zhao, Ziyu, et autres
Publié: (2025)

Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models
par: Son, Hyegang, et autres
Publié: (2024)

RevFFN: Memory-Efficient Full-Parameter Fine-Tuning of Mixture-of-Experts LLMs with Reversible Blocks
par: Liu, Ningyuan, et autres
Publié: (2025)

Ensembles of Low-Rank Expert Adapters
par: Li, Yinghao, et autres
Publié: (2025)

RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation
par: Liu, Jun, et autres
Publié: (2025)

Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
par: Liu, Yilun, et autres
Publié: (2025)

BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching
par: Cong, Peizhuang, et autres
Publié: (2024)

Spectral Adapter: Fine-Tuning in Spectral Space
par: Zhang, Fangzhao, et autres
Publié: (2024)

MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
par: Raje, Arian, et autres
Publié: (2026)

OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs
par: Gao, Yuting, et autres
Publié: (2025)

Aggregating Low Rank Adapters in Federated Fine-tuning
par: Trautmann, Evelyn, et autres
Publié: (2025)

Enhancing Talent Search Ranking with Role-Aware Expert Mixtures and LLM-based Fine-Grained Job Descriptions
par: Li, Jihang, et autres
Publié: (2025)

Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in $\{\pm 1, \pm i\}$
par: Wang, Feiyu, et autres
Publié: (2025)

MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
par: Yang, Cheng, et autres
Publié: (2024)

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
par: Modoranu, Ionut-Vlad, et autres
Publié: (2026)

Memory-Efficient Fine-Tuning via Low-Rank Activation Compression
par: Shi, Jiang-Xin, et autres
Publié: (2025)

Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesis
par: Li, Weikai, et autres
Publié: (2024)

Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts
par: Zhang, Buze, et autres
Publié: (2026)

TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
par: Kunwar, Pradip, et autres
Publié: (2025)

SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning
par: Xie, Zhen-Hao, et autres
Publié: (2026)

Non-Neighbors Also Matter to Kriging: A New Contrastive-Prototypical Learning
par: Li, Zhishuai, et autres
Publié: (2024)

INT-FlashAttention: Enabling Flash Attention for INT8 Quantization
par: Chen, Shimao, et autres
Publié: (2024)

MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
par: Zeng, Runjia, et autres
Publié: (2025)

L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
par: Yang, Minghao, et autres
Publié: (2026)

MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
par: Tang, Chuanyu, et autres
Publié: (2024)

Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
par: Hu, Wentao, et autres
Publié: (2025)

Sparse High Rank Adapters
par: Bhardwaj, Kartikeya, et autres
Publié: (2024)

Efficiently Editing Mixture-of-Experts Models with Compressed Experts
par: He, Yifei, et autres
Publié: (2025)

HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
par: Wei, Jia, et autres
Publié: (2026)

AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers
par: Khazem, Salim
Publié: (2026)

Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLM
par: Codefuse, et autres
Publié: (2025)

Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
par: Kim, Gyeongman, et autres
Publié: (2025)

AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert
par: Gao, Yuting, et autres
Publié: (2025)

ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models
par: Singhal, Raghav, et autres
Publié: (2025)

MoKA: Mixture of Kronecker Adapters
par: Sadeghi, Mohammadreza, et autres
Publié: (2025)