Surrogate Benchmarks for Model Merging Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916930253750272 |
|---|---|
| author | Akizuki, Rio Kudo, Yuya Yoshinari, Nozomu Hirose, Yoichi Nishimoto, Toshiyuki Uchida, Kento Shirakawa, Shinichi |
| author_facet | Akizuki, Rio Kudo, Yuya Yoshinari, Nozomu Hirose, Yoichi Nishimoto, Toshiyuki Uchida, Kento Shirakawa, Shinichi |
| contents | Model merging techniques aim to integrate the abilities of multiple models into a single model. Most model merging techniques have hyperparameters, and their setting affects the performance of the merged model. Because several existing works show that tuning hyperparameters in model merging can enhance the merging outcome, developing hyperparameter optimization algorithms for model merging is a promising direction. However, its optimization process is computationally expensive, particularly in merging LLMs. In this work, we develop surrogate benchmarks for optimization of the merging hyperparameters to realize algorithm development and performance comparison at low cost. We define two search spaces and collect data samples to construct surrogate models to predict the performance of a merged model from a hyperparameter. We demonstrate that our benchmarks can predict the performance of merged models well and simulate optimization algorithm behaviors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_02555 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Surrogate Benchmarks for Model Merging Optimization Akizuki, Rio Kudo, Yuya Yoshinari, Nozomu Hirose, Yoichi Nishimoto, Toshiyuki Uchida, Kento Shirakawa, Shinichi Machine Learning Artificial Intelligence Neural and Evolutionary Computing Model merging techniques aim to integrate the abilities of multiple models into a single model. Most model merging techniques have hyperparameters, and their setting affects the performance of the merged model. Because several existing works show that tuning hyperparameters in model merging can enhance the merging outcome, developing hyperparameter optimization algorithms for model merging is a promising direction. However, its optimization process is computationally expensive, particularly in merging LLMs. In this work, we develop surrogate benchmarks for optimization of the merging hyperparameters to realize algorithm development and performance comparison at low cost. We define two search spaces and collect data samples to construct surrogate models to predict the performance of a merged model from a hyperparameter. We demonstrate that our benchmarks can predict the performance of merged models well and simulate optimization algorithm behaviors. |
| title | Surrogate Benchmarks for Model Merging Optimization |
| topic | Machine Learning Artificial Intelligence Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2509.02555 |