PruneCD: Contrasting Pruned Self Model to Improve Decoding Factuality
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916963265019904 |
|---|---|
| author | Yu, Byeongho Lee, Changhun Jin, Jungyu Park, Eunhyeok |
| author_facet | Yu, Byeongho Lee, Changhun Jin, Jungyu Park, Eunhyeok |
| contents | To mitigate the hallucination problem in large language models, DoLa exploits early exit logits from the same model as a contrastive prior. However, we found that these early exit logits tend to be flat, low in magnitude, and fail to reflect meaningful contrasts. To address this, we propose PruneCD, a novel contrastive decoding method that constructs the amateur model via layer pruning rather than early exit. This design leads to more informative and well-aligned logits, enabling more effective contrastive decoding. Through qualitative and quantitative analyses, we demonstrate that PruneCD consistently improves factuality with minimal inference overhead, offering a robust and practical approach to mitigating hallucinations in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_16598 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PruneCD: Contrasting Pruned Self Model to Improve Decoding Factuality Yu, Byeongho Lee, Changhun Jin, Jungyu Park, Eunhyeok Computation and Language Artificial Intelligence To mitigate the hallucination problem in large language models, DoLa exploits early exit logits from the same model as a contrastive prior. However, we found that these early exit logits tend to be flat, low in magnitude, and fail to reflect meaningful contrasts. To address this, we propose PruneCD, a novel contrastive decoding method that constructs the amateur model via layer pruning rather than early exit. This design leads to more informative and well-aligned logits, enabling more effective contrastive decoding. Through qualitative and quantitative analyses, we demonstrate that PruneCD consistently improves factuality with minimal inference overhead, offering a robust and practical approach to mitigating hallucinations in LLMs. |
| title | PruneCD: Contrasting Pruned Self Model to Improve Decoding Factuality |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.16598 |