Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915154287919104 |
|---|---|
| author | Nguyen, Hieu He, Zihao Gandre, Shoumik Atul Pasupulety, Ujjwal Shivakumar, Sharanya Kumari Lerman, Kristina |
| author_facet | Nguyen, Hieu He, Zihao Gandre, Shoumik Atul Pasupulety, Ujjwal Shivakumar, Sharanya Kumari Lerman, Kristina |
| contents | Large language models (LLMs) often suffer from hallucination, generating factually incorrect or ungrounded content, which limits their reliability in high-stakes applications. A key factor contributing to hallucination is the use of hard labels during training, which enforce deterministic supervision, encourage overconfidence, and disregard the uncertainty inherent in natural language. To address this, we propose mitigating hallucination through knowledge distillation (KD), where a teacher model provides smoothed soft labels to a student model, reducing overconfidence and improving factual grounding. We apply KD during supervised finetuning on instructional data, evaluating its effectiveness across LLMs from different families. Experimental results on summarization benchmarks demonstrate that KD reduces hallucination compared to standard finetuning while preserving performance on general NLP tasks. These findings highlight KD as a promising approach for mitigating hallucination in LLMs and improving model reliability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11306 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillation Nguyen, Hieu He, Zihao Gandre, Shoumik Atul Pasupulety, Ujjwal Shivakumar, Sharanya Kumari Lerman, Kristina Computation and Language Machine Learning Large language models (LLMs) often suffer from hallucination, generating factually incorrect or ungrounded content, which limits their reliability in high-stakes applications. A key factor contributing to hallucination is the use of hard labels during training, which enforce deterministic supervision, encourage overconfidence, and disregard the uncertainty inherent in natural language. To address this, we propose mitigating hallucination through knowledge distillation (KD), where a teacher model provides smoothed soft labels to a student model, reducing overconfidence and improving factual grounding. We apply KD during supervised finetuning on instructional data, evaluating its effectiveness across LLMs from different families. Experimental results on summarization benchmarks demonstrate that KD reduces hallucination compared to standard finetuning while preserving performance on general NLP tasks. These findings highlight KD as a promising approach for mitigating hallucination in LLMs and improving model reliability. |
| title | Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillation |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.11306 |