Cover Image

What Matters for Model Merging at Scale?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yadav, Prateek, Vu, Tu, Lai, Jonathan, Chronopoulou, Alexandra, Faruqui, Manaal, Bansal, Mohit, Munkhdalai, Tsendsuren
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
by: Munkhdalai, Tsendsuren, et al.
Published: (2024)

Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation
by: Vu, Tu, et al.
Published: (2024)

ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
by: Yadav, Prateek, et al.
Published: (2023)

Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy
by: Li, Pingzhi, et al.
Published: (2023)

RSQ: Learning from Important Tokens Leads to Better Quantized LLMs
by: Sung, Yi-Lin, et al.
Published: (2025)

Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
by: Saha, Swarnadeep, et al.
Published: (2023)

A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
by: Yadav, Prateek, et al.
Published: (2024)

Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models
by: Munkhdalai, Tsendsuren, et al.
Published: (2024)

Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings
by: Jiang, Yichen, et al.
Published: (2024)

Soft Self-Consistency Improves Language Model Agents
by: Wang, Han, et al.
Published: (2024)

SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
by: Li, Jialu, et al.
Published: (2024)

Multi-Attribute Steering of Language Models via Targeted Intervention
by: Nguyen, Duy, et al.
Published: (2025)

UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning
by: Patil, Vaidehi, et al.
Published: (2025)

DAM: Dynamic Adapter Merging for Continual Video QA Learning
by: Cheng, Feng, et al.
Published: (2024)

ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
by: Chen, Justin Chih-Yao, et al.
Published: (2023)

The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
by: Hase, Peter, et al.
Published: (2024)

One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Exploration
by: Khan, Zaid, et al.
Published: (2025)

DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback
by: Khan, Zaid, et al.
Published: (2024)

Merging by Matching Models in Task Parameter Subspaces
by: Tam, Derek, et al.
Published: (2023)

Merge to Mix: Mixing Datasets via Model Merging
by: Tao, Zhixu Silvia, et al.
Published: (2025)

Effective Reasoning Chains Reduce Intrinsic Dimensionality
by: Prasad, Archiki, et al.
Published: (2026)

EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
by: Zala, Abhay, et al.
Published: (2024)

Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
by: Xiao, Hanqi, et al.
Published: (2025)

Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models
by: Prasad, Archiki, et al.
Published: (2023)

How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not
by: Verdini, Francesco, et al.
Published: (2024)

ADaPT: As-Needed Decomposition and Planning with Language Models
by: Prasad, Archiki, et al.
Published: (2023)

ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models
by: Sung, Yi-Lin, et al.
Published: (2023)

Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging
by: Lu, Zhenyi, et al.
Published: (2024)

MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies
by: Zhang, Shiyue, et al.
Published: (2023)

Executable Functional Abstractions: Inferring Generative Programs for Advanced Math Problems
by: Khan, Zaid, et al.
Published: (2025)

PRInTS: Reward Modeling for Long-Horizon Information Seeking
by: Lee, Jaewoo, et al.
Published: (2025)

ReGAL: Refactoring Programs to Discover Generalizable Abstractions
by: Stengel-Eskin, Elias, et al.
Published: (2024)

Arcee's MergeKit: A Toolkit for Merging Large Language Models
by: Goddard, Charles, et al.
Published: (2024)

System-1.x: Learning to Balance Fast and Slow Planning with Language Models
by: Saha, Swarnadeep, et al.
Published: (2024)

Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation
by: Glocker, Kevin, et al.
Published: (2025)

Model Merging for Knowledge Editing
by: Fu, Zichuan, et al.
Published: (2025)

K-Merge: Online Continual Merging of Adapters for On-device Large Language Models
by: Shenaj, Donald, et al.
Published: (2025)

Efficient Model Development through Fine-tuning Transfer
by: Lin, Pin-Jie, et al.
Published: (2025)

Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
by: Yang, Jinluan, et al.
Published: (2025)

Dynamic Model Merging Made Slim
by: Du, Guodong, et al.
Published: (2026)