Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.09834 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916896376356864 |
|---|---|
| author | Sun, Weigao Hu, Jiaxi Zhou, Yucheng Du, Jusen Lan, Disen Wang, Kexin Zhu, Tong Qu, Xiaoye Zhang, Yu Mo, Xiaoyu Liu, Daizong Liang, Yuxuan Chen, Wenliang Li, Guoqi Cheng, Yu |
| author_facet | Sun, Weigao Hu, Jiaxi Zhou, Yucheng Du, Jusen Lan, Disen Wang, Kexin Zhu, Tong Qu, Xiaoye Zhang, Yu Mo, Xiaoyu Liu, Daizong Liang, Yuxuan Chen, Wenliang Li, Guoqi Cheng, Yu |
| contents | Large Language Models (LLMs) have delivered impressive results in language understanding, generation, reasoning, and pushes the ability boundary of multimodal models. Transformer models, as the foundation of modern LLMs, offer a strong baseline with excellent scaling properties. However, the traditional transformer architecture requires substantial computations and poses significant obstacles for large-scale training and practical deployment. In this survey, we offer a systematic examination of innovative LLM architectures that address the inherent limitations of transformers and boost the efficiency. Starting from language modeling, this survey covers the background and technical details of linear and sparse sequence modeling methods, efficient full attention variants, sparse mixture-of-experts, hybrid model architectures incorporating the above techniques, and emerging diffusion LLMs. Additionally, we discuss applications of these techniques to other modalities and consider their wider implications for developing scalable, resource-aware foundation models. By grouping recent studies into the above category, this survey presents a blueprint of modern efficient LLM architectures, and we hope this could help motivate future research toward more efficient, versatile AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09834 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Speed Always Wins: A Survey on Efficient Architectures for Large Language Models Sun, Weigao Hu, Jiaxi Zhou, Yucheng Du, Jusen Lan, Disen Wang, Kexin Zhu, Tong Qu, Xiaoye Zhang, Yu Mo, Xiaoyu Liu, Daizong Liang, Yuxuan Chen, Wenliang Li, Guoqi Cheng, Yu Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Large Language Models (LLMs) have delivered impressive results in language understanding, generation, reasoning, and pushes the ability boundary of multimodal models. Transformer models, as the foundation of modern LLMs, offer a strong baseline with excellent scaling properties. However, the traditional transformer architecture requires substantial computations and poses significant obstacles for large-scale training and practical deployment. In this survey, we offer a systematic examination of innovative LLM architectures that address the inherent limitations of transformers and boost the efficiency. Starting from language modeling, this survey covers the background and technical details of linear and sparse sequence modeling methods, efficient full attention variants, sparse mixture-of-experts, hybrid model architectures incorporating the above techniques, and emerging diffusion LLMs. Additionally, we discuss applications of these techniques to other modalities and consider their wider implications for developing scalable, resource-aware foundation models. By grouping recent studies into the above category, this survey presents a blueprint of modern efficient LLM architectures, and we hope this could help motivate future research toward more efficient, versatile AI systems. |
| title | Speed Always Wins: A Survey on Efficient Architectures for Large Language Models |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.09834 |