Cover Image

Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Shen, Li, Tang, Anke, Yang, Enneng, Guo, Guibing, Luo, Yong, Zhang, Lefei, Cao, Xiaochun, Du, Bo, Tao, Dacheng
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computer Vision and Pattern Recognition
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Merging Multi-Task Models via Weight-Ensembling Mixture of Experts
by: Tang, Anke, et al.
Published: (2024)

AdaMerging: Adaptive Model Merging for Multi-Task Learning
by: Yang, Enneng, et al.
Published: (2023)

Representation Surgery for Multi-Task Model Merging
by: Yang, Enneng, et al.
Published: (2024)

Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
by: Yang, Enneng, et al.
Published: (2024)

Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
by: Tang, Anke, et al.
Published: (2025)

SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery
by: Yang, Enneng, et al.
Published: (2024)

FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion
by: Tang, Anke, et al.
Published: (2024)

SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models
by: Tang, Anke, et al.
Published: (2024)

OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging
by: Wei, Yongxian, et al.
Published: (2025)

Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion
by: Tang, Anke, et al.
Published: (2024)

Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent
by: Wei, Yongxian, et al.
Published: (2025)

Generalized Embedding Machines for Recommender Systems
by: Yang, Enneng, et al.
Published: (2020)

Parameter Efficient Multi-task Model Fusion with Partial Linearization
by: Tang, Anke, et al.
Published: (2023)

Continual Learning From a Stream of APIs
by: Yang, Enneng, et al.
Published: (2023)

Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion
by: Lan, Pengxiang, et al.
Published: (2024)

Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Product
by: Lan, Pengxiang, et al.
Published: (2025)

Causal Direct Preference Optimization for Distributionally Robust Generative Recommendation
by: Zhao, Chu, et al.
Published: (2026)

AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
by: Yang, Enneng, et al.
Published: (2022)

Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer
by: Kong, Yilun, et al.
Published: (2025)

ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learning
by: Zhao, Chu, et al.
Published: (2026)

Hard Negative Sampling via Large Language Models for Recommendation
by: Zhao, Chu, et al.
Published: (2025)

Repeated Padding+: Simple yet Effective Data Augmentation Plugin for Sequential Recommendation
by: Dang, Yizhou, et al.
Published: (2024)

DeGMix: Efficient Multi-Task Dense Prediction with Deformable and Gating Mixer
by: Xu, Yangyang, et al.
Published: (2023)

Learning from models beyond fine-tuning
by: Zheng, Hongling, et al.
Published: (2023)

Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace
by: Yang, Jinluan, et al.
Published: (2024)

Multi-Task Dense Prediction via Mixture of Low-Rank Experts
by: Yang, Yuqi, et al.
Published: (2024)

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
by: Liu, Boan, et al.
Published: (2023)

Distributionally Robust Graph Out-of-Distribution Recommendation via Diffusion Model
by: Zhao, Chu, et al.
Published: (2025)

Symmetric Graph Contrastive Learning against Noisy Views for Recommendation
by: Zhao, Chu, et al.
Published: (2024)

MMGRid: Navigating Temporal-aware and Cross-domain Generative Recommendation via Model Merging
by: Wei, Tianjun, et al.
Published: (2026)

Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
by: Li, Lujun, et al.
Published: (2025)

Generalizable and Efficient Automated Scoring with a Knowledge-Distilled Multi-Task Mixture-of-Experts
by: Fang, Luyang, et al.
Published: (2025)

Learning More Generalized Experts by Merging Experts in Mixture-of-Experts
by: Park, Sejik
Published: (2024)

CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging
by: Cao, Jie, et al.
Published: (2026)

Multi-Task Model Merging via Adaptive Weight Disentanglement
by: Xiong, Feng, et al.
Published: (2024)

Expert Merging in Sparse Mixture of Experts with Nash Bargaining
by: Nguyen, Dung V., et al.
Published: (2025)

Data Augmentation for Sequential Recommendation: A Survey
by: Dang, Yizhou, et al.
Published: (2024)

Data Augmentation as Free Lunch: Exploring the Test-Time Augmentation for Sequential Recommendation
by: Dang, Yizhou, et al.
Published: (2025)

Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts
by: Lotfi, Sanae, et al.
Published: (2026)

Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
by: Guo, Yongxin, et al.
Published: (2024)