Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909395174031360 |
|---|---|
| author | Ren, Ruiyang Wang, Yuhao Qu, Yingqi Zhao, Wayne Xin Liu, Jing Tian, Hao Wu, Hua Wen, Ji-Rong Wang, Haifeng |
| author_facet | Ren, Ruiyang Wang, Yuhao Qu, Yingqi Zhao, Wayne Xin Liu, Jing Tian, Hao Wu, Hua Wen, Ji-Rong Wang, Haifeng |
| contents | Large language models (LLMs) have shown impressive prowess in solving a wide range of tasks with world knowledge. However, it remains unclear how well LLMs are able to perceive their factual knowledge boundaries, particularly under retrieval augmentation settings. In this study, we present the first analysis on the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain question answering (QA), with a bunch of important findings. Specifically, we focus on three research questions and analyze them by examining QA, priori judgement and posteriori judgement capabilities of LLMs. We show evidence that LLMs possess unwavering confidence in their knowledge and cannot handle the conflict between internal and external knowledge well. Furthermore, retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We further conduct thorough experiments to examine how different factors affect LLMs and propose a simple method to dynamically utilize supporting documents with our judgement strategy. Additionally, we find that the relevance between the supporting documents and the questions significantly impacts LLMs' QA and judgemental capabilities. The code to reproduce this work is available at https://github.com/RUCAIBox/LLM-Knowledge-Boundary. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_11019 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation Ren, Ruiyang Wang, Yuhao Qu, Yingqi Zhao, Wayne Xin Liu, Jing Tian, Hao Wu, Hua Wen, Ji-Rong Wang, Haifeng Computation and Language Information Retrieval Large language models (LLMs) have shown impressive prowess in solving a wide range of tasks with world knowledge. However, it remains unclear how well LLMs are able to perceive their factual knowledge boundaries, particularly under retrieval augmentation settings. In this study, we present the first analysis on the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain question answering (QA), with a bunch of important findings. Specifically, we focus on three research questions and analyze them by examining QA, priori judgement and posteriori judgement capabilities of LLMs. We show evidence that LLMs possess unwavering confidence in their knowledge and cannot handle the conflict between internal and external knowledge well. Furthermore, retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We further conduct thorough experiments to examine how different factors affect LLMs and propose a simple method to dynamically utilize supporting documents with our judgement strategy. Additionally, we find that the relevance between the supporting documents and the questions significantly impacts LLMs' QA and judgemental capabilities. The code to reproduce this work is available at https://github.com/RUCAIBox/LLM-Knowledge-Boundary. |
| title | Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2307.11019 |