Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2511.12869 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917222761365504 |
|---|---|
| author | Mohsin, Muhammad Ahmed Umer, Muhammad Bilal, Ahsan Memon, Zeeshan Qadir, Muhammad Ibtsaam Bhattacharya, Sagnik Rizwan, Hassan Gorle, Abhiram R. Kazmi, Maahe Zehra Amir, Nukhba Subhan, Ali Rafique, Muhammad Usman He, Zihao Mehta, Pulkit Jamshed, Muhammad Ali Cioffi, John M. |
| author_facet | Mohsin, Muhammad Ahmed Umer, Muhammad Bilal, Ahsan Memon, Zeeshan Qadir, Muhammad Ibtsaam Bhattacharya, Sagnik Rizwan, Hassan Gorle, Abhiram R. Kazmi, Maahe Zehra Amir, Nukhba Subhan, Ali Rafique, Muhammad Usman He, Zihao Mehta, Pulkit Jamshed, Muhammad Ali Cioffi, John M. |
| contents | Large Language Models (LLMs) have benefited enormously from scaling, yet these gains are bounded by five fundamental limitations: (1) hallucination, (2) context compression, (3) reasoning degradation, (4) retrieval fragility, and (5) multimodal misalignment. While existing surveys describe these phenomena empirically, they lack a rigorous theoretical synthesis connecting them to the foundational limits of computation, information, and learning. This work closes that gap by presenting a unified, proof-informed framework that formalizes the innate theoretical ceilings of LLM scaling. First, computability and uncomputability imply an irreducible residue of error: for any computably enumerable model family, diagonalization guarantees inputs on which some model must fail, and undecidable queries (e.g., halting-style tasks) induce infinite failure sets for all computable predictors. Second, information-theoretic and statistical constraints bound attainable accuracy even on decidable tasks, finite description length enforces compression error, and long-tail factual knowledge requires prohibitive sample complexity. Third, geometric and computational effects compress long contexts far below their nominal size due to positional under-training, encoding attenuation, and softmax crowding. We further show how likelihood-based training favors pattern completion over inference, how retrieval under token limits suffers from semantic drift and coupling noise, and how multimodal scaling inherits shallow cross-modal alignment. Across sections, we pair theorems and empirical evidence to outline where scaling helps, where it saturates, and where it cannot progress, providing both theoretical foundations and practical mitigation paths like bounded-oracle retrieval, positional curricula, and sparse or hierarchical attention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12869 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Fundamental Limits of LLMs at Scale Mohsin, Muhammad Ahmed Umer, Muhammad Bilal, Ahsan Memon, Zeeshan Qadir, Muhammad Ibtsaam Bhattacharya, Sagnik Rizwan, Hassan Gorle, Abhiram R. Kazmi, Maahe Zehra Amir, Nukhba Subhan, Ali Rafique, Muhammad Usman He, Zihao Mehta, Pulkit Jamshed, Muhammad Ali Cioffi, John M. Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Information Theory Multiagent Systems Large Language Models (LLMs) have benefited enormously from scaling, yet these gains are bounded by five fundamental limitations: (1) hallucination, (2) context compression, (3) reasoning degradation, (4) retrieval fragility, and (5) multimodal misalignment. While existing surveys describe these phenomena empirically, they lack a rigorous theoretical synthesis connecting them to the foundational limits of computation, information, and learning. This work closes that gap by presenting a unified, proof-informed framework that formalizes the innate theoretical ceilings of LLM scaling. First, computability and uncomputability imply an irreducible residue of error: for any computably enumerable model family, diagonalization guarantees inputs on which some model must fail, and undecidable queries (e.g., halting-style tasks) induce infinite failure sets for all computable predictors. Second, information-theoretic and statistical constraints bound attainable accuracy even on decidable tasks, finite description length enforces compression error, and long-tail factual knowledge requires prohibitive sample complexity. Third, geometric and computational effects compress long contexts far below their nominal size due to positional under-training, encoding attenuation, and softmax crowding. We further show how likelihood-based training favors pattern completion over inference, how retrieval under token limits suffers from semantic drift and coupling noise, and how multimodal scaling inherits shallow cross-modal alignment. Across sections, we pair theorems and empirical evidence to outline where scaling helps, where it saturates, and where it cannot progress, providing both theoretical foundations and practical mitigation paths like bounded-oracle retrieval, positional curricula, and sparse or hierarchical attention. |
| title | On the Fundamental Limits of LLMs at Scale |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Information Theory Multiagent Systems |
| url | https://arxiv.org/abs/2511.12869 |