Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.02907 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915855856566272 |
|---|---|
| author | Gan, Zeyu Ren, Ruifeng Yao, Wei Hu, Xiaolin Xu, Gengze Qian, Chen Tang, Huayi Gong, Zixuan Yao, Xinhao Tang, Pengwei Dou, Zhenxing Liu, Yong |
| author_facet | Gan, Zeyu Ren, Ruifeng Yao, Wei Hu, Xiaolin Xu, Gengze Qian, Chen Tang, Huayi Gong, Zixuan Yao, Xinhao Tang, Pengwei Dou, Zhenxing Liu, Yong |
| contents | The rapid emergence of Large Language Models (LLMs) has precipitated a profound paradigm shift in Artificial Intelligence, delivering monumental engineering successes that increasingly impact modern society. However, a critical paradox persists within the current field: despite the empirical efficacy, our theoretical understanding of LLMs remains disproportionately nascent, forcing these systems to be treated largely as ``black boxes''. To address this theoretical fragmentation, this survey proposes a unified lifecycle-based taxonomy that organizes the research landscape into six distinct stages: Data Preparation, Model Preparation, Training, Alignment, Inference, and Evaluation. Within this framework, we provide a systematic review of the foundational theories and internal mechanisms driving LLM performance. Specifically, we analyze core theoretical issues such as the mathematical justification for data mixtures, the representational limits of various architectures, and the optimization dynamics of alignment algorithms. Moving beyond current best practices, we identify critical frontier challenges, including the theoretical limits of synthetic data self-improvement, the mathematical bounds of safety guarantees, and the mechanistic origins of emergent intelligence. By connecting empirical observations with rigorous scientific inquiry, this work provides a structured roadmap for transitioning LLM development from engineering heuristics toward a principled scientific discipline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_02907 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond the Black Box: A Survey on the Theory and Mechanism of Large Language Models Gan, Zeyu Ren, Ruifeng Yao, Wei Hu, Xiaolin Xu, Gengze Qian, Chen Tang, Huayi Gong, Zixuan Yao, Xinhao Tang, Pengwei Dou, Zhenxing Liu, Yong Computation and Language The rapid emergence of Large Language Models (LLMs) has precipitated a profound paradigm shift in Artificial Intelligence, delivering monumental engineering successes that increasingly impact modern society. However, a critical paradox persists within the current field: despite the empirical efficacy, our theoretical understanding of LLMs remains disproportionately nascent, forcing these systems to be treated largely as ``black boxes''. To address this theoretical fragmentation, this survey proposes a unified lifecycle-based taxonomy that organizes the research landscape into six distinct stages: Data Preparation, Model Preparation, Training, Alignment, Inference, and Evaluation. Within this framework, we provide a systematic review of the foundational theories and internal mechanisms driving LLM performance. Specifically, we analyze core theoretical issues such as the mathematical justification for data mixtures, the representational limits of various architectures, and the optimization dynamics of alignment algorithms. Moving beyond current best practices, we identify critical frontier challenges, including the theoretical limits of synthetic data self-improvement, the mathematical bounds of safety guarantees, and the mechanistic origins of emergent intelligence. By connecting empirical observations with rigorous scientific inquiry, this work provides a structured roadmap for transitioning LLM development from engineering heuristics toward a principled scientific discipline. |
| title | Beyond the Black Box: A Survey on the Theory and Mechanism of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.02907 |