The Entropy Limit of Generative Intelligence
Fuente:
Zenodo
Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Recurso digital |
| Pubblicazione: |
Zenodo
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866901113651855360 |
|---|---|
| author | Troche, Elias |
| author_facet | Troche, Elias |
| contents | <p>Large language models exhibit a range of degradation phenomena — hallucination, catastrophic forgetting, lost in the middle, alignment drift — that the industry treats as separate problems requiring separate solutions. We propose that these phenomena are unified manifestations of a single underlying process: informational error accumulation in high-complexity, low-decoupling systems operating without active correction mechanisms.</p> <p>We introduce the Informational Persistence Model (IPM), a constraint framework derived from the thermodynamics of information processing, and operationalize it for AI systems through four measurable parameters: metabolic intensity (η), informational complexity (K), error-correction fidelity (c), and structural decoupling (S). The IPM predicts that model degradation is constrained by a deterministic boundary determined by the persistence coefficient Φ = (c·S)/(η·K).</p> <p>We validate the framework retrospectively against published degradation data from five established studies: TruthfulQA (hallucination scaling), Liu et al. 2023 (lost in the middle), Kirkpatrick et al. 2017 (catastrophic forgetting), Anthropic sleeper agents (alignment drift), and Mixtral MoE comparisons. We calculate Φ for two representative architectures — Llama-3-70B (dense transformer) and DeepSeek-V3 (Mixture of Experts) — demonstrating that the MoE architecture achieves significantly higher persistence due to its structural decoupling (S = 0.65 vs 0.15).</p> <p>We propose five falsifiable predictions and five immediate applications — from production monitoring to architectural licensing — that any organization can implement with existing tools. We conclude that current scaling practices — increasing K and η without proportional increases in c and S — are thermodynamically unsustainable. The useful lifespan of generative models is not a matter of engineering failure but of physical constraint.</p> <p><em> </em></p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_19355037 |
| institution | Zenodo |
| language | |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | The Entropy Limit of Generative Intelligence Troche, Elias Informational Persistence Large Language Models Model Degradation Catastrophic Forgetting Thermodynamics of Computation Production Monitoring AI Reliability Structural Decoupling <p>Large language models exhibit a range of degradation phenomena — hallucination, catastrophic forgetting, lost in the middle, alignment drift — that the industry treats as separate problems requiring separate solutions. We propose that these phenomena are unified manifestations of a single underlying process: informational error accumulation in high-complexity, low-decoupling systems operating without active correction mechanisms.</p> <p>We introduce the Informational Persistence Model (IPM), a constraint framework derived from the thermodynamics of information processing, and operationalize it for AI systems through four measurable parameters: metabolic intensity (η), informational complexity (K), error-correction fidelity (c), and structural decoupling (S). The IPM predicts that model degradation is constrained by a deterministic boundary determined by the persistence coefficient Φ = (c·S)/(η·K).</p> <p>We validate the framework retrospectively against published degradation data from five established studies: TruthfulQA (hallucination scaling), Liu et al. 2023 (lost in the middle), Kirkpatrick et al. 2017 (catastrophic forgetting), Anthropic sleeper agents (alignment drift), and Mixtral MoE comparisons. We calculate Φ for two representative architectures — Llama-3-70B (dense transformer) and DeepSeek-V3 (Mixture of Experts) — demonstrating that the MoE architecture achieves significantly higher persistence due to its structural decoupling (S = 0.65 vs 0.15).</p> <p>We propose five falsifiable predictions and five immediate applications — from production monitoring to architectural licensing — that any organization can implement with existing tools. We conclude that current scaling practices — increasing K and η without proportional increases in c and S — are thermodynamically unsustainable. The useful lifespan of generative models is not a matter of engineering failure but of physical constraint.</p> <p><em> </em></p> |
| title | The Entropy Limit of Generative Intelligence |
| topic | Informational Persistence Large Language Models Model Degradation Catastrophic Forgetting Thermodynamics of Computation Production Monitoring AI Reliability Structural Decoupling |
| url | https://doi.org/10.5281/zenodo.19355037 |