Learning or Self-aligning? Rethinking Instruction Fine-tuning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914908002582528 |
|---|---|
| author | Ren, Mengjie Cao, Boxi Lin, Hongyu Liu, Cao Han, Xianpei Zeng, Ke Wan, Guanglu Cai, Xunliang Sun, Le |
| author_facet | Ren, Mengjie Cao, Boxi Lin, Hongyu Liu, Cao Han, Xianpei Zeng, Ke Wan, Guanglu Cai, Xunliang Sun, Le |
| contents | Instruction Fine-tuning~(IFT) is a critical phase in building large language models~(LLMs). Previous works mainly focus on the IFT's role in the transfer of behavioral norms and the learning of additional world knowledge. However, the understanding of the underlying mechanisms of IFT remains significantly limited. In this paper, we design a knowledge intervention framework to decouple the potential underlying factors of IFT, thereby enabling individual analysis of different factors. Surprisingly, our experiments reveal that attempting to learn additional world knowledge through IFT often struggles to yield positive impacts and can even lead to markedly negative effects. Further, we discover that maintaining internal knowledge consistency before and after IFT is a critical factor for achieving successful IFT. Our findings reveal the underlying mechanisms of IFT and provide robust support for some very recent and potential future works. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_18243 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning or Self-aligning? Rethinking Instruction Fine-tuning Ren, Mengjie Cao, Boxi Lin, Hongyu Liu, Cao Han, Xianpei Zeng, Ke Wan, Guanglu Cai, Xunliang Sun, Le Computation and Language Instruction Fine-tuning~(IFT) is a critical phase in building large language models~(LLMs). Previous works mainly focus on the IFT's role in the transfer of behavioral norms and the learning of additional world knowledge. However, the understanding of the underlying mechanisms of IFT remains significantly limited. In this paper, we design a knowledge intervention framework to decouple the potential underlying factors of IFT, thereby enabling individual analysis of different factors. Surprisingly, our experiments reveal that attempting to learn additional world knowledge through IFT often struggles to yield positive impacts and can even lead to markedly negative effects. Further, we discover that maintaining internal knowledge consistency before and after IFT is a critical factor for achieving successful IFT. Our findings reveal the underlying mechanisms of IFT and provide robust support for some very recent and potential future works. |
| title | Learning or Self-aligning? Rethinking Instruction Fine-tuning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.18243 |