On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909622648963072 |
|---|---|
| author | Wen, Xueru Lou, Jie Lu, Xinyu Yuqiu, Ji Guan, Xinyan Lu, Yaojie Lin, Hongyu He, Ben Han, Xianpei Zhang, Debing Sun, Le |
| author_facet | Wen, Xueru Lou, Jie Lu, Xinyu Yuqiu, Ji Guan, Xinyan Lu, Yaojie Lin, Hongyu He, Ben Han, Xianpei Zhang, Debing Sun, Le |
| contents | Hallucination occurs when large language models exhibit behavior that deviates from the boundaries of their knowledge during response generation. To address this critical issue, previous learning-based methods attempt to finetune models but are limited by off-policy sampling and coarse-grained feedback. In this paper, we present \textit{\b{R}einforcement \b{L}earning \b{f}or \b{H}allucination} (RLFH), an on-policy self-alignment approach that enables LLMs to actively explore their knowledge boundaries and self-correct generation behavior through fine-grained feedback signals. RLFH introduces a self-assessment framework where the policy serves as its own judge. Through this framework, responses are automatically decomposed into atomic facts and their truthfulness and informativeness are assessed against external knowledge sources. The resulting fine-grained feedback at the statement level are then converted into token-level dense reward signals. This enables online reinforcement learning to achieve precise and timely optimization without human intervention. Comprehensive evaluations on HotpotQA, SQuADv2, and Biography benchmarks validate RLFH's effectiveness in hallucination mitigation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12221 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation Wen, Xueru Lou, Jie Lu, Xinyu Yuqiu, Ji Guan, Xinyan Lu, Yaojie Lin, Hongyu He, Ben Han, Xianpei Zhang, Debing Sun, Le Computation and Language Hallucination occurs when large language models exhibit behavior that deviates from the boundaries of their knowledge during response generation. To address this critical issue, previous learning-based methods attempt to finetune models but are limited by off-policy sampling and coarse-grained feedback. In this paper, we present \textit{\b{R}einforcement \b{L}earning \b{f}or \b{H}allucination} (RLFH), an on-policy self-alignment approach that enables LLMs to actively explore their knowledge boundaries and self-correct generation behavior through fine-grained feedback signals. RLFH introduces a self-assessment framework where the policy serves as its own judge. Through this framework, responses are automatically decomposed into atomic facts and their truthfulness and informativeness are assessed against external knowledge sources. The resulting fine-grained feedback at the statement level are then converted into token-level dense reward signals. This enables online reinforcement learning to achieve precise and timely optimization without human intervention. Comprehensive evaluations on HotpotQA, SQuADv2, and Biography benchmarks validate RLFH's effectiveness in hallucination mitigation. |
| title | On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.12221 |