Why Fine-Tuning Encourages Hallucinations and How to Fix It
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917415772749824 |
|---|---|
| author | Kaplan, Guy Gekhman, Zorik Zhu, Zhen Rozner, Lotem Reif, Yuval Swayamdipta, Swabha Hoiem, Derek Schwartz, Roy |
| author_facet | Kaplan, Guy Gekhman, Zorik Zhu, Zhen Rozner, Lotem Reif, Yuval Swayamdipta, Swabha Hoiem, Derek Schwartz, Roy |
| contents | Large language models are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning (SFT), which can increase hallucinations w.r.t. knowledge acquired during pre-training. In this work, we explore whether SFT-induced hallucinations can be mitigated using established tools from the continual learning literature, since they arise as a by-product of knowledge degradation during training. We propose a self-distillation-based SFT method that facilitates effective factual learning while minimizing hallucinations w.r.t. pre-existing knowledge by regularizing output-distribution drift. We also show that, in settings where new knowledge acquisition is unnecessary, suppressing factual plasticity by freezing parameter groups, can preserve task performance while reducing hallucinations. Lastly, we investigate the mechanism behind SFT-induced hallucinations through three hypotheses: capacity limitations, behavior cloning, and localized interference. Our experiments show that a main driver is interference among overlapping semantic representations, and that self-distillation succeeds by mitigating this interference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_15574 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Why Fine-Tuning Encourages Hallucinations and How to Fix It Kaplan, Guy Gekhman, Zorik Zhu, Zhen Rozner, Lotem Reif, Yuval Swayamdipta, Swabha Hoiem, Derek Schwartz, Roy Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing Large language models are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning (SFT), which can increase hallucinations w.r.t. knowledge acquired during pre-training. In this work, we explore whether SFT-induced hallucinations can be mitigated using established tools from the continual learning literature, since they arise as a by-product of knowledge degradation during training. We propose a self-distillation-based SFT method that facilitates effective factual learning while minimizing hallucinations w.r.t. pre-existing knowledge by regularizing output-distribution drift. We also show that, in settings where new knowledge acquisition is unnecessary, suppressing factual plasticity by freezing parameter groups, can preserve task performance while reducing hallucinations. Lastly, we investigate the mechanism behind SFT-induced hallucinations through three hypotheses: capacity limitations, behavior cloning, and localized interference. Our experiments show that a main driver is interference among overlapping semantic representations, and that self-distillation succeeds by mitigating this interference. |
| title | Why Fine-Tuning Encourages Hallucinations and How to Fix It |
| topic | Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2604.15574 |