How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916524344737792 |
|---|---|
| author | Zheng, Danna Lapata, Mirella Pan, Jeff Z. |
| author_facet | Zheng, Danna Lapata, Mirella Pan, Jeff Z. |
| contents | Large Language Models (LLMs) are increasingly explored as knowledge bases (KBs), yet current evaluation methods focus too narrowly on knowledge retention, overlooking other crucial criteria for reliable performance. In this work, we rethink the requirements for evaluating reliable LLM-as-KB usage and highlight two essential factors: factuality, ensuring accurate responses to seen and unseen knowledge, and consistency, maintaining stable answers to questions about the same knowledge. We introduce UnseenQA, a dataset designed to assess LLM performance on unseen knowledge, and propose new criteria and metrics to quantify factuality and consistency, leading to a final reliability score. Our experiments on 26 LLMs reveal several challenges regarding their use as KBs, underscoring the need for more principled and comprehensive evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_13578 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency Zheng, Danna Lapata, Mirella Pan, Jeff Z. Computation and Language Artificial Intelligence Large Language Models (LLMs) are increasingly explored as knowledge bases (KBs), yet current evaluation methods focus too narrowly on knowledge retention, overlooking other crucial criteria for reliable performance. In this work, we rethink the requirements for evaluating reliable LLM-as-KB usage and highlight two essential factors: factuality, ensuring accurate responses to seen and unseen knowledge, and consistency, maintaining stable answers to questions about the same knowledge. We introduce UnseenQA, a dataset designed to assess LLM performance on unseen knowledge, and propose new criteria and metrics to quantify factuality and consistency, leading to a final reliability score. Our experiments on 26 LLMs reveal several challenges regarding their use as KBs, underscoring the need for more principled and comprehensive evaluation. |
| title | How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.13578 |