PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918319050719232 |
|---|---|
| author | Liu, Langming Lv, Kangtao Chen, Haibin Zhang, Weidong Wang, Yejing Liu, Shilei Tong, Xin Yuan, Yujin Wang, Yongwei Su, Wenbo Zheng, Bo |
| author_facet | Liu, Langming Lv, Kangtao Chen, Haibin Zhang, Weidong Wang, Yejing Liu, Shilei Tong, Xin Yuan, Yujin Wang, Yongwei Su, Wenbo Zheng, Bo |
| contents | Large language models (LLMs), despite their powerful capabilities, suffer from factual hallucinations where they generate verifiable falsehoods. We identify a root of this issue: the imbalanced data distribution in the pretraining corpus, which leads to a state of "low-probability truth" and "high-probability falsehood". Recent approaches, such as teaching models to say "I don't know" or post-hoc knowledge editing, either evade the problem or face catastrophic forgetting. To address this issue from its root, we propose \textbf{PretrainRL}, a novel framework that integrates reinforcement learning into the pretraining phase to consolidate factual knowledge. The core principle of PretrainRL is "\textbf{debiasing then learning}." It actively reshapes the model's probability distribution by down-weighting high-probability falsehoods, thereby making "room" for low-probability truths to be learned effectively. To enable this, we design an efficient negative sampling strategy to discover these high-probability falsehoods and introduce novel metrics to evaluate the model's probabilistic state concerning factual knowledge. Extensive experiments on three public benchmarks demonstrate that PretrainRL significantly alleviates factual hallucinations and outperforms state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01875 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginning Liu, Langming Lv, Kangtao Chen, Haibin Zhang, Weidong Wang, Yejing Liu, Shilei Tong, Xin Yuan, Yujin Wang, Yongwei Su, Wenbo Zheng, Bo Computation and Language Large language models (LLMs), despite their powerful capabilities, suffer from factual hallucinations where they generate verifiable falsehoods. We identify a root of this issue: the imbalanced data distribution in the pretraining corpus, which leads to a state of "low-probability truth" and "high-probability falsehood". Recent approaches, such as teaching models to say "I don't know" or post-hoc knowledge editing, either evade the problem or face catastrophic forgetting. To address this issue from its root, we propose \textbf{PretrainRL}, a novel framework that integrates reinforcement learning into the pretraining phase to consolidate factual knowledge. The core principle of PretrainRL is "\textbf{debiasing then learning}." It actively reshapes the model's probability distribution by down-weighting high-probability falsehoods, thereby making "room" for low-probability truths to be learned effectively. To enable this, we design an efficient negative sampling strategy to discover these high-probability falsehoods and introduce novel metrics to evaluate the model's probabilistic state concerning factual knowledge. Extensive experiments on three public benchmarks demonstrate that PretrainRL significantly alleviates factual hallucinations and outperforms state-of-the-art methods. |
| title | PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.01875 |