Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.07024 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912637684547584 |
|---|---|
| author | Ghosh, Shrestha Giordano, Luca Hu, Yujia Nguyen, Tuan-Phong Razniewski, Simon |
| author_facet | Ghosh, Shrestha Giordano, Luca Hu, Yujia Nguyen, Tuan-Phong Razniewski, Simon |
| contents | LLMs are remarkable artifacts that have revolutionized a range of NLP and AI tasks. A significant contributor is their factual knowledge, which, to date, remains poorly understood, and is usually analyzed from biased samples. In this paper, we take a deep tour into the factual knowledge (or beliefs) of a frontier LLM, based on GPTKB v1.5 (Hu et al., 2025a), a recursively elicited set of 100 million beliefs of one of the strongest currently available frontier LLMs, GPT-4.1. We find that the models' factual knowledge differs quite significantly from established knowledge bases, and that its accuracy is significantly lower than indicated by previous benchmarks. We also find that inconsistency, ambiguity and hallucinations are major issues, shedding light on future research opportunities concerning factual LLM knowledge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07024 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mining the Mind: What 100M Beliefs Reveal About Frontier LLM Knowledge Ghosh, Shrestha Giordano, Luca Hu, Yujia Nguyen, Tuan-Phong Razniewski, Simon Computation and Language Artificial Intelligence LLMs are remarkable artifacts that have revolutionized a range of NLP and AI tasks. A significant contributor is their factual knowledge, which, to date, remains poorly understood, and is usually analyzed from biased samples. In this paper, we take a deep tour into the factual knowledge (or beliefs) of a frontier LLM, based on GPTKB v1.5 (Hu et al., 2025a), a recursively elicited set of 100 million beliefs of one of the strongest currently available frontier LLMs, GPT-4.1. We find that the models' factual knowledge differs quite significantly from established knowledge bases, and that its accuracy is significantly lower than indicated by previous benchmarks. We also find that inconsistency, ambiguity and hallucinations are major issues, shedding light on future research opportunities concerning factual LLM knowledge. |
| title | Mining the Mind: What 100M Beliefs Reveal About Frontier LLM Knowledge |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.07024 |