MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Merging
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911414920151040 |
|---|---|
| author | Kong, Fanshuang Zhang, Richong Nie, Zhijie Zhou, Hang Wang, Ziqiao Sun, Qiang Hu, Chunming |
| author_facet | Kong, Fanshuang Zhang, Richong Nie, Zhijie Zhou, Hang Wang, Ziqiao Sun, Qiang Hu, Chunming |
| contents | Model merging offers a scalable alternative to multi-task learning but often yields suboptimal performance on classification tasks. We attribute this degradation to a geometric misalignment between the merged encoder and static task-specific classifier heads. Existing methods typically rely on auxiliary parameters to enforce strict representation alignment. We challenge this approach by revealing that the misalignment is predominantly an orthogonal transformation, rendering such strict alignment unnecessary. Leveraging this insight, we propose MOMA (Masked Orthogonal Matrix Alignment), which rectifies the misalignment by jointly optimizing a global multi-task vector mask and task-specific orthogonal transformations. Crucially, MOMA absorbs corresponding new parameters directly into the existing model weights, achieving performance comparable to state-of-the-art baselines with zero additional parameters and zero added inference cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13526 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Merging Kong, Fanshuang Zhang, Richong Nie, Zhijie Zhou, Hang Wang, Ziqiao Sun, Qiang Hu, Chunming Machine Learning Model merging offers a scalable alternative to multi-task learning but often yields suboptimal performance on classification tasks. We attribute this degradation to a geometric misalignment between the merged encoder and static task-specific classifier heads. Existing methods typically rely on auxiliary parameters to enforce strict representation alignment. We challenge this approach by revealing that the misalignment is predominantly an orthogonal transformation, rendering such strict alignment unnecessary. Leveraging this insight, we propose MOMA (Masked Orthogonal Matrix Alignment), which rectifies the misalignment by jointly optimizing a global multi-task vector mask and task-specific orthogonal transformations. Crucially, MOMA absorbs corresponding new parameters directly into the existing model weights, achieving performance comparable to state-of-the-art baselines with zero additional parameters and zero added inference cost. |
| title | MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Merging |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2412.13526 |