Cover Image

SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Tang, Anke, Shen, Li, Luo, Yong, Xie, Shuai, Hu, Han, Zhang, Lefei, Du, Bo, Tao, Dacheng
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Merging Multi-Task Models via Weight-Ensembling Mixture of Experts
by: Tang, Anke, et al.
Published: (2024)

Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
by: Shen, Li, et al.
Published: (2024)

FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion
by: Tang, Anke, et al.
Published: (2024)

Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion
by: Tang, Anke, et al.
Published: (2024)

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
by: Sun, Mengyang, et al.
Published: (2025)

Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
by: Tang, Anke, et al.
Published: (2025)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
by: Song, Weixi, et al.
Published: (2023)

Multi-Task Dense Prediction via Mixture of Low-Rank Experts
by: Yang, Yuqi, et al.
Published: (2024)

OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models
by: Wang, Shuai, et al.
Published: (2024)

Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs
by: Hu, Zixuan, et al.
Published: (2024)

Learning from models beyond fine-tuning
by: Zheng, Hongling, et al.
Published: (2023)

Parameter Efficient Multi-task Model Fusion with Partial Linearization
by: Tang, Anke, et al.
Published: (2023)

From Sparse to Soft Mixtures of Experts
by: Puigcerver, Joan, et al.
Published: (2023)

FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
by: Pan, Xinglin, et al.
Published: (2025)

MoPE: Mixture of Prefix Experts for Zero-Shot Dialogue State Tracking
by: Tang, Tianwen, et al.
Published: (2024)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
by: Pan, Bowen, et al.
Published: (2024)

Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation
by: Luo, Xiang, et al.
Published: (2024)

Specializing Foundation Models via Mixture of Low-Rank Experts for Comprehensive Head CT Analysis
by: Yoo, Youngjin, et al.
Published: (2026)

RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
by: Sun, Mengyang, et al.
Published: (2026)

Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Experts
by: Nikolic, Strahinja, et al.
Published: (2025)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
by: Belenki, Lior, et al.
Published: (2025)

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
by: Liu, Boan, et al.
Published: (2023)

Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
by: Panda, Ashwinee, et al.
Published: (2025)

Training Sparse Mixture Of Experts Text Embedding Models
by: Nussbaum, Zach, et al.
Published: (2025)

MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
by: Tang, Chuanyu, et al.
Published: (2024)

Intention Analysis Makes LLMs A Good Jailbreak Defender
by: Zhang, Yuqi, et al.
Published: (2024)

Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalization
by: Hu, Shengchao, et al.
Published: (2024)

Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers
by: Pavlitska, Svetlana, et al.
Published: (2025)

HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptation
by: Lei, Qinqian, et al.
Published: (2025)

Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
by: Zhao, Ziyu, et al.
Published: (2025)

Low-Rank Mixture-of-Experts for Continual Medical Image Segmentation
by: Chen, Qian, et al.
Published: (2024)

MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
by: Jin, Peng, et al.
Published: (2024)

MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classification
by: Li, Yapeng, et al.
Published: (2025)

MLLM-Guided VLM Fine-Tuning with Joint Inference for Zero-Shot Composed Image Retrieval
by: Tu, Rong-Cheng, et al.
Published: (2025)

L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts
by: Ji, Shihao, et al.
Published: (2025)

CtrlAttack: A Unified Attack on World-Model Control in Diffusion Models
by: Xu, Shuhan, et al.
Published: (2026)

Rethink Sparse Signals for Pose-guided Text-to-image Generation
by: Xuan, Wenjie, et al.
Published: (2025)

Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer
by: Kong, Yilun, et al.
Published: (2025)

EVA: Mixture-of-Experts Semantic Variant Alignment for Compositional Zero-Shot Learning
by: Zhang, Xiao, et al.
Published: (2025)

AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts
by: Liu, Zefang, et al.
Published: (2024)