On the Fundamental Impossibility of Hallucination Control in Large Language Models
Fuente:
arXiv
Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909848981995520 |
|---|---|
| author | Karpowicz, Michał P. |
| author_facet | Karpowicz, Michał P. |
| contents | This paper establishes a fundamental Impossibility Theorem: no LLM performing non-trivial knowledge aggregation can simultaneously achieve truthful knowledge representation, semantic information conservation, complete revelation of relevant knowledge, and knowledge-constrained optimality. This impossibility stems from the mathematical structure of information aggregation, not from engineering limitations.
We prove this by modeling inference as an auction of ideas, where distributed components compete to influence responses using their encoded knowledge. The proof employs three independent approaches: mechanism design (Green-Laffont theorem), proper scoring rules (Savage), and transformer architecture analysis (log-sum-exp convexity).
We introduce the semantic information measure and the emergence operator to analyze computationally bounded and unbounded reasoning. Bounded reasoning makes latent information accessible, enabling gradual insights and creativity, while unbounded reasoning makes all derivable knowledge immediately accessible while preserving the semantic content. We prove the conservation-reasoning dichotomy: meaningful reasoning necessarily violates information conservation.
Our framework suggests that hallucination and imagination are mathematically identical, and both violate at least one of the four essential properties. The Jensen gap in transformer attention quantifies this violation as excess confidence beyond constituent evidence. This unified view explains why capable models must balance truthfulness against creativity.
These results provide principled foundations for managing hallucination trade-offs in AI systems. Rather than eliminating hallucination, we should optimize these inevitable trade-offs for specific applications. We conclude with philosophical implications connecting the impossibility to fundamental limits of reason. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06382 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Fundamental Impossibility of Hallucination Control in Large Language Models Karpowicz, Michał P. Machine Learning Artificial Intelligence Computation and Language Computer Science and Game Theory This paper establishes a fundamental Impossibility Theorem: no LLM performing non-trivial knowledge aggregation can simultaneously achieve truthful knowledge representation, semantic information conservation, complete revelation of relevant knowledge, and knowledge-constrained optimality. This impossibility stems from the mathematical structure of information aggregation, not from engineering limitations. We prove this by modeling inference as an auction of ideas, where distributed components compete to influence responses using their encoded knowledge. The proof employs three independent approaches: mechanism design (Green-Laffont theorem), proper scoring rules (Savage), and transformer architecture analysis (log-sum-exp convexity). We introduce the semantic information measure and the emergence operator to analyze computationally bounded and unbounded reasoning. Bounded reasoning makes latent information accessible, enabling gradual insights and creativity, while unbounded reasoning makes all derivable knowledge immediately accessible while preserving the semantic content. We prove the conservation-reasoning dichotomy: meaningful reasoning necessarily violates information conservation. Our framework suggests that hallucination and imagination are mathematically identical, and both violate at least one of the four essential properties. The Jensen gap in transformer attention quantifies this violation as excess confidence beyond constituent evidence. This unified view explains why capable models must balance truthfulness against creativity. These results provide principled foundations for managing hallucination trade-offs in AI systems. Rather than eliminating hallucination, we should optimize these inevitable trade-offs for specific applications. We conclude with philosophical implications connecting the impossibility to fundamental limits of reason. |
| title | On the Fundamental Impossibility of Hallucination Control in Large Language Models |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Science and Game Theory |
| url | https://arxiv.org/abs/2506.06382 |