Cover Image

MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Jiapeng, Tian, Changxin, Chen, Kunlong, Liu, Ziqi, Mao, Jiaxin, Zhao, Wayne Xin, Zhang, Zhiqiang, Zhou, Jun
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
by: Tian, Changxin, et al.
Published: (2025)

MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics
by: Wang, Jiapeng, et al.
Published: (2025)

Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
by: Tian, Changxin, et al.
Published: (2025)

MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding
by: Jin, Xin, et al.
Published: (2025)

Arrows of Math Reasoning Data Synthesis for Large Language Models: Diversity, Complexity and Correctness
by: Chen, Sirui, et al.
Published: (2025)

Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
by: Yang, Jinluan, et al.
Published: (2025)

BOSE: A Systematic Evaluation Method Optimized for Base Models
by: Luan, Hongzhi, et al.
Published: (2025)

Merge to Mix: Mixing Datasets via Model Merging
by: Tao, Zhixu Silvia, et al.
Published: (2025)

FW-Merging: Scaling Model Merging with Frank-Wolfe Optimization
by: Chen, Hao Mark, et al.
Published: (2025)

RV-Syn: Rational and Verifiable Mathematical Reasoning Data Synthesis based on Structured Function Library
by: Wang, Jiapeng, et al.
Published: (2025)

FedMerge: Federated Personalization via Model Merging
by: Chen, Shutong, et al.
Published: (2025)

DC-Merge: Improving Model Merging with Directional Consistency
by: Zhang, Han-Chen, et al.
Published: (2026)

Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learning
by: Aakanksha, et al.
Published: (2024)

Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization
by: Berasi, Davide, et al.
Published: (2026)

Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training
by: Li, Shengrui, et al.
Published: (2026)

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
by: Zhou, Wenjie, et al.
Published: (2026)

Efficient Visual Transformer by Learnable Token Merging
by: Wang, Yancheng, et al.
Published: (2024)

Model Merging in the Essential Subspace
by: Li, Longhua, et al.
Published: (2026)

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
by: Chen, Weiyu, et al.
Published: (2024)

Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging
by: Bertolissi, Ryo, et al.
Published: (2025)

Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts
by: Morrison, Jacob, et al.
Published: (2026)

Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data
by: Zhang, Bingjie, et al.
Published: (2025)

Do Not Merge My Model! Safeguarding Open-Source LLMs Against Unauthorized Model Merging
by: Li, Qinfeng, et al.
Published: (2025)

PSO-Merging: Merging Models Based on Particle Swarm Optimization
by: Zhang, Kehao, et al.
Published: (2025)

Training-Free Model Merging for Multi-target Domain Adaptation
by: Li, Wenyi, et al.
Published: (2024)

BadMerging: Backdoor Attacks Against Model Merging
by: Zhang, Jinghuai, et al.
Published: (2024)

Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data
by: Wang, Jiapeng, et al.
Published: (2025)

MergeMoE: Efficient Compression of MoE Models via Expert Output Merging
by: Miao, Ruijie, et al.
Published: (2025)

Multi-task Code LLMs: Data Mix or Model Merge?
by: Zhu, Mingzhi, et al.
Published: (2026)

MIN-Merging: Merge the Important Neurons for Model Merging
by: Liang, Yunfei
Published: (2025)

AutoMerge: Search-Based Model Merging Framework for Effective Model Reuse
by: Lu, You, et al.
Published: (2026)

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
by: Sun, Wenju, et al.
Published: (2025)

Merging Multi-Task Models via Weight-Ensembling Mixture of Experts
by: Tang, Anke, et al.
Published: (2024)

Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression
by: Gao, Junqi, et al.
Published: (2026)

ATM: Improving Model Merging by Alternating Tuning and Merging
by: Zhou, Luca, et al.
Published: (2024)

SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging
by: Chen, Zijun, et al.
Published: (2025)

Training-Free Pretrained Model Merging
by: Xu, Zhengqi, et al.
Published: (2024)

Training-free Heterogeneous Model Merging
by: Xu, Zhengqi, et al.
Published: (2024)

ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation
by: Xu, Bo, et al.
Published: (2026)

Adapting Multilingual Models to Code-Mixed Tasks via Model Merging
by: Kodali, Prashant, et al.
Published: (2025)