UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915312607166464 |
|---|---|
| author | Tan, Chuanyuan Shao, Wenbiao Xiong, Hao Zhu, Tong Liu, Zhenhua Shi, Kai Chen, Wenliang |
| author_facet | Tan, Chuanyuan Shao, Wenbiao Xiong, Hao Zhu, Tong Liu, Zhenhua Shi, Kai Chen, Wenliang |
| contents | Handling unanswerable questions (UAQ) is crucial for LLMs, as it helps prevent misleading responses in complex situations. While previous studies have built several datasets to assess LLMs' performance on UAQ, these datasets lack factual knowledge support, which limits the evaluation of LLMs' ability to utilize their factual knowledge when handling UAQ. To address the limitation, we introduce a new unanswerable question dataset UAQFact, a bilingual dataset with auxiliary factual knowledge created from a Knowledge Graph. Based on UAQFact, we further define two new tasks to measure LLMs' ability to utilize internal and external factual knowledge, respectively. Our experimental results across multiple LLM series show that UAQFact presents significant challenges, as LLMs do not consistently perform well even when they have factual knowledge stored. Additionally, we find that incorporating external knowledge may enhance performance, but LLMs still cannot make full use of the knowledge which may result in incorrect responses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23461 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions Tan, Chuanyuan Shao, Wenbiao Xiong, Hao Zhu, Tong Liu, Zhenhua Shi, Kai Chen, Wenliang Computation and Language Handling unanswerable questions (UAQ) is crucial for LLMs, as it helps prevent misleading responses in complex situations. While previous studies have built several datasets to assess LLMs' performance on UAQ, these datasets lack factual knowledge support, which limits the evaluation of LLMs' ability to utilize their factual knowledge when handling UAQ. To address the limitation, we introduce a new unanswerable question dataset UAQFact, a bilingual dataset with auxiliary factual knowledge created from a Knowledge Graph. Based on UAQFact, we further define two new tasks to measure LLMs' ability to utilize internal and external factual knowledge, respectively. Our experimental results across multiple LLM series show that UAQFact presents significant challenges, as LLMs do not consistently perform well even when they have factual knowledge stored. Additionally, we find that incorporating external knowledge may enhance performance, but LLMs still cannot make full use of the knowledge which may result in incorrect responses. |
| title | UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.23461 |