Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.11403 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911003469414400 |
|---|---|
| author | Ritter-Gutierrez, Fabian Lin, Yi-Cheng Wong, Jeremy H. M Lee, Hung-yi Chng, Eng Siong Chen, Nancy F. |
| author_facet | Ritter-Gutierrez, Fabian Lin, Yi-Cheng Wong, Jeremy H. M Lee, Hung-yi Chng, Eng Siong Chen, Nancy F. |
| contents | Creating a unified speech and music model requires expensive pre-training. Model merging can instead create an unified audio model with minimal computational expense. However, direct merging is challenging when the models are not aligned in the weight space. Motivated by Git Re-Basin, we introduce a correlation-permutation approach that aligns a music encoder's internal layers with a speech encoder. We extend previous work to the case of merging transformer layers. The method computes a permutation matrix that maximizes the model's features-wise cross-correlations layer by layer, enabling effective fusion of these otherwise disjoint models. The merged model retains speech capabilities through this method while significantly enhancing music performance, achieving an improvement of 14.83 points in average score compared to linear interpolation model merging. This work allows the creation of unified audio models from independently trained encoders. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_11403 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A correlation-permutation approach for speech-music encoders model merging Ritter-Gutierrez, Fabian Lin, Yi-Cheng Wong, Jeremy H. M Lee, Hung-yi Chng, Eng Siong Chen, Nancy F. Sound Artificial Intelligence Audio and Speech Processing Creating a unified speech and music model requires expensive pre-training. Model merging can instead create an unified audio model with minimal computational expense. However, direct merging is challenging when the models are not aligned in the weight space. Motivated by Git Re-Basin, we introduce a correlation-permutation approach that aligns a music encoder's internal layers with a speech encoder. We extend previous work to the case of merging transformer layers. The method computes a permutation matrix that maximizes the model's features-wise cross-correlations layer by layer, enabling effective fusion of these otherwise disjoint models. The merged model retains speech capabilities through this method while significantly enhancing music performance, achieving an improvement of 14.83 points in average score compared to linear interpolation model merging. This work allows the creation of unified audio models from independently trained encoders. |
| title | A correlation-permutation approach for speech-music encoders model merging |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2506.11403 |