Chained Tuning Leads to Biased Forgetting
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916541879025664 |
|---|---|
| author | Ung, Megan Sun, Alicia Bell, Samuel J. Radharapu, Bhaktipriya Sagun, Levent Williams, Adina |
| author_facet | Ung, Megan Sun, Alicia Bell, Samuel J. Radharapu, Bhaktipriya Sagun, Levent Williams, Adina |
| contents | Large language models (LLMs) are often fine-tuned for use on downstream tasks, though this can degrade capabilities learned during previous training. This phenomenon, often referred to as catastrophic forgetting, has important potential implications for the safety of deployed models. In this work, we first show that models trained on downstream tasks forget their safety tuning to a greater extent than models trained in the opposite order. Second, we show that forgetting disproportionately impacts safety information about certain groups. To quantify this phenomenon, we define a new metric we term biased forgetting. We conduct a systematic evaluation of the effects of task ordering on forgetting and apply mitigations that can help the model recover from the forgetting observed. We hope our findings can better inform methods for chaining the finetuning of LLMs in continual learning settings to enable training of safer and less toxic models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_16469 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Chained Tuning Leads to Biased Forgetting Ung, Megan Sun, Alicia Bell, Samuel J. Radharapu, Bhaktipriya Sagun, Levent Williams, Adina Computation and Language Large language models (LLMs) are often fine-tuned for use on downstream tasks, though this can degrade capabilities learned during previous training. This phenomenon, often referred to as catastrophic forgetting, has important potential implications for the safety of deployed models. In this work, we first show that models trained on downstream tasks forget their safety tuning to a greater extent than models trained in the opposite order. Second, we show that forgetting disproportionately impacts safety information about certain groups. To quantify this phenomenon, we define a new metric we term biased forgetting. We conduct a systematic evaluation of the effects of task ordering on forgetting and apply mitigations that can help the model recover from the forgetting observed. We hope our findings can better inform methods for chaining the finetuning of LLMs in continual learning settings to enable training of safer and less toxic models. |
| title | Chained Tuning Leads to Biased Forgetting |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.16469 |