Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929353718235136 |
|---|---|
| author | Fu, Tingchen Cai, Deng Liu, Lemao Shi, Shuming Yan, Rui |
| author_facet | Fu, Tingchen Cai, Deng Liu, Lemao Shi, Shuming Yan, Rui |
| contents | Supervised fine-tuning (SFT) on instruction-following corpus is a crucial approach toward the alignment of large language models (LLMs). However, the performance of LLMs on standard knowledge and reasoning benchmarks tends to suffer from deterioration at the latter stage of the SFT process, echoing the phenomenon of alignment tax. Through our pilot study, we put a hypothesis that the data biases are probably one cause behind the phenomenon. To address the issue, we introduce a simple disperse-then-merge framework. To be concrete, we disperse the instruction-following data into portions and train multiple sub-models using different data portions. Then we merge multiple models into a single one via model merging techniques. Despite its simplicity, our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_13432 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction Fu, Tingchen Cai, Deng Liu, Lemao Shi, Shuming Yan, Rui Computation and Language Artificial Intelligence Supervised fine-tuning (SFT) on instruction-following corpus is a crucial approach toward the alignment of large language models (LLMs). However, the performance of LLMs on standard knowledge and reasoning benchmarks tends to suffer from deterioration at the latter stage of the SFT process, echoing the phenomenon of alignment tax. Through our pilot study, we put a hypothesis that the data biases are probably one cause behind the phenomenon. To address the issue, we introduce a simple disperse-then-merge framework. To be concrete, we disperse the instruction-following data into portions and train multiple sub-models using different data portions. Then we merge multiple models into a single one via model merging techniques. Despite its simplicity, our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks. |
| title | Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.13432 |