Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914290560139264 |
|---|---|
| author | Yun, Sungmin Park, Seonyong Nam, Hwayong Lee, Younjoo Lee, Gunjun Kyung, Kwanhee Kim, Sangpyo Kim, Nam Sung Kim, Jongmin Kim, Hyungyo Cho, Juhwan Baek, Seungmin Ahn, Jung Ho |
| author_facet | Yun, Sungmin Park, Seonyong Nam, Hwayong Lee, Younjoo Lee, Gunjun Kyung, Kwanhee Kim, Sangpyo Kim, Nam Sung Kim, Jongmin Kim, Hyungyo Cho, Juhwan Baek, Seungmin Ahn, Jung Ho |
| contents | Computational workloads composing traditional transformer models are starkly bifurcated. Multi-Head Attention (MHA) and Grouped-Query Attention are memory-bound due to low arithmetic intensity, while FeedForward Networks are compute-bound. This dichotomy has long motivated research into specialized hardware to mitigate the attention bottleneck.
This paper argues that recent architectural advances in transformer models -- Multi-head Latent Attention (MLA) and Mixture of Experts (MoE) -- introduce new dominant bottlenecks, shifting the challenge away from memory-intensive attention. We make two key observations. First, the arithmetic intensity of MLA is over two orders of magnitude higher than that of MHA, moving it toward a compute-bound regime well-matched to modern accelerators such as GPUs. Second, distributing MoE experts across a pool of accelerators allows batching to tune their arithmetic intensity to that of dense layers, producing a more balanced computational profile. Consequently, the focus of hardware and system optimization should shift from attention acceleration to high-bandwidth interconnects and balancing expert workloads across accelerators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15465 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts Yun, Sungmin Park, Seonyong Nam, Hwayong Lee, Younjoo Lee, Gunjun Kyung, Kwanhee Kim, Sangpyo Kim, Nam Sung Kim, Jongmin Kim, Hyungyo Cho, Juhwan Baek, Seungmin Ahn, Jung Ho Hardware Architecture Artificial Intelligence Computational workloads composing traditional transformer models are starkly bifurcated. Multi-Head Attention (MHA) and Grouped-Query Attention are memory-bound due to low arithmetic intensity, while FeedForward Networks are compute-bound. This dichotomy has long motivated research into specialized hardware to mitigate the attention bottleneck. This paper argues that recent architectural advances in transformer models -- Multi-head Latent Attention (MLA) and Mixture of Experts (MoE) -- introduce new dominant bottlenecks, shifting the challenge away from memory-intensive attention. We make two key observations. First, the arithmetic intensity of MLA is over two orders of magnitude higher than that of MHA, moving it toward a compute-bound regime well-matched to modern accelerators such as GPUs. Second, distributing MoE experts across a pool of accelerators allows batching to tune their arithmetic intensity to that of dense layers, producing a more balanced computational profile. Consequently, the focus of hardware and system optimization should shift from attention acceleration to high-bandwidth interconnects and balancing expert workloads across accelerators. |
| title | Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts |
| topic | Hardware Architecture Artificial Intelligence |
| url | https://arxiv.org/abs/2507.15465 |