Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909352812609536 |
|---|---|
| author | Morrison, Jacob Smith, Noah A. Hajishirzi, Hannaneh Koh, Pang Wei Dodge, Jesse Dasigi, Pradeep |
| author_facet | Morrison, Jacob Smith, Noah A. Hajishirzi, Hannaneh Koh, Pang Wei Dodge, Jesse Dasigi, Pradeep |
| contents | Adapting general-purpose language models to new skills is currently an expensive process that must be repeated as new instruction datasets targeting new skills are created, or can cause the models to forget older skills. In this work, we investigate the effectiveness of adding new skills to preexisting models by training on the new skills in isolation and later merging with the general model (e.g. using task vectors). In experiments focusing on scientific literature understanding, safety, and coding, we find that the parallel-train-then-merge procedure, which is significantly cheaper than retraining the models on updated data mixtures, is often comparably effective. Our experiments also show that parallel training is especially well-suited for enabling safety features in LMs relative to continued finetuning and retraining, as it dramatically improves model compliance with safe prompts while preserving its ability to refuse dangerous or harmful prompts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12937 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging Morrison, Jacob Smith, Noah A. Hajishirzi, Hannaneh Koh, Pang Wei Dodge, Jesse Dasigi, Pradeep Computation and Language Machine Learning Adapting general-purpose language models to new skills is currently an expensive process that must be repeated as new instruction datasets targeting new skills are created, or can cause the models to forget older skills. In this work, we investigate the effectiveness of adding new skills to preexisting models by training on the new skills in isolation and later merging with the general model (e.g. using task vectors). In experiments focusing on scientific literature understanding, safety, and coding, we find that the parallel-train-then-merge procedure, which is significantly cheaper than retraining the models on updated data mixtures, is often comparably effective. Our experiments also show that parallel training is especially well-suited for enabling safety features in LMs relative to continued finetuning and retraining, as it dramatically improves model compliance with safe prompts while preserving its ability to refuse dangerous or harmful prompts. |
| title | Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2410.12937 |