Cover Image

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhou, Wenjie, Wang, Bohan, Zhang, Hongtao, Jia, Chenxi, Chen, Wei, Cheng, Xueqi
Format:	Preprint
Published:	2026
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training
by: Zhang, Hongtao, et al.
Published: (2026)

BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Training
by: Zhou, Wenjie, et al.
Published: (2025)

Model Merging in the Essential Subspace
by: Li, Longhua, et al.
Published: (2026)

Model Merging in Pre-training of Large Language Models
by: Li, Yunshui, et al.
Published: (2025)

When and Why Grouping Attention Heads Accelerates Muon Optimization
by: Zhang, Hongtao, et al.
Published: (2026)

Subspace-Boosted Model Merging
by: Skorobogat, Ronald, et al.
Published: (2025)

FedMerge: Federated Personalization via Model Merging
by: Chen, Shutong, et al.
Published: (2025)

Merging by Matching Models in Task Parameter Subspaces
by: Tam, Derek, et al.
Published: (2023)

MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
by: Wang, Jiapeng, et al.
Published: (2026)

Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging
by: Zhang, Haobo, et al.
Published: (2025)

DC-Merge: Improving Model Merging with Directional Consistency
by: Zhang, Han-Chen, et al.
Published: (2026)

Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging
by: Lu, Zhenyi, et al.
Published: (2024)

Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging
by: Morrison, Jacob, et al.
Published: (2024)

Bridging Domains through Subspace-Aware Model Merging
by: Chaves, Levy, et al.
Published: (2026)

SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models
by: Reza, Md Kaykobad, et al.
Published: (2026)

Channel Merging: Preserving Specialization for Merged Experts
by: Zhang, Mingyang, et al.
Published: (2024)

MIN-Merging: Merge the Important Neurons for Model Merging
by: Liang, Yunfei
Published: (2025)

SeMe: Training-Free Language Model Merging via Semantic Alignment
by: Gu, Jian, et al.
Published: (2025)

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
by: Marczak, Daniel, et al.
Published: (2025)

Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
by: Yang, Jinluan, et al.
Published: (2025)

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
by: Sun, Wenju, et al.
Published: (2025)

Training-free Heterogeneous Model Merging
by: Xu, Zhengqi, et al.
Published: (2024)

Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging
by: Chen, Weiyu, et al.
Published: (2024)

BadMerging: Backdoor Attacks Against Model Merging
by: Zhang, Jinghuai, et al.
Published: (2024)

FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision-Language Models
by: Huang, Chenyu, et al.
Published: (2026)

Merge and Guide: Unifying Model Merging and Guided Decoding for Controllable Multi-Objective Generation
by: Xie, Guofu, et al.
Published: (2025)

PSO-Merging: Merging Models Based on Particle Swarm Optimization
by: Zhang, Kehao, et al.
Published: (2025)

Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace
by: Yang, Jinluan, et al.
Published: (2024)

Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data
by: Zhang, Bingjie, et al.
Published: (2025)

Merge to Mix: Mixing Datasets via Model Merging
by: Tao, Zhixu Silvia, et al.
Published: (2025)

Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking
by: Zhang, Dengming, et al.
Published: (2025)

Rethinking Layer-wise Model Merging through Chain of Merges
by: Buzzega, Pietro, et al.
Published: (2025)

Orthogonal Model Merging
by: Yang, Sihan, et al.
Published: (2026)

FW-Merging: Scaling Model Merging with Frank-Wolfe Optimization
by: Chen, Hao Mark, et al.
Published: (2025)

Dataless Knowledge Fusion by Merging Weights of Language Models
by: Jin, Xisen, et al.
Published: (2022)

Arcee's MergeKit: A Toolkit for Merging Large Language Models
by: Goddard, Charles, et al.
Published: (2024)

AdaRank: Adaptive Rank Pruning for Enhanced Model Merging
by: Lee, Chanhyuk, et al.
Published: (2025)

NAN: A Training-Free Solution to Coefficient Estimation in Model Merging
by: Si, Chongjie, et al.
Published: (2025)

Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning
by: Zhou, Da-Wei, et al.
Published: (2024)

ATM: Improving Model Merging by Alternating Tuning and Merging
by: Zhou, Luca, et al.
Published: (2024)