The Anatomy of a Triton Attention Kernel
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911266331688960 |
|---|---|
| author | Ringlein, Burkhard van Lunteren, Jan Stoica, Radu Parnell, Thomas |
| author_facet | Ringlein, Burkhard van Lunteren, Jan Stoica, Radu Parnell, Thomas |
| contents | A long-standing goal in both industry and academia is to develop an LLM inference platform that is portable across hardware architectures, eliminates the need for low-level hand-tuning, and still delivers best-in-class efficiency. In this work, we demonstrate that portable, efficient cross-platform LLM inference is indeed possible and share our experience. We develop a state-of-the-art paged attention kernel, the core performance-critical component of many LLM deployments, that builds exclusively on the domain-specific just-in-time compiled language Triton to achieve state-of-the-art performance on both NVIDIA and AMD GPUs. We describe our high-level approach, the key algorithmic and system-level improvements, the parameter auto-tuning required to unlock efficiency, and the integrations into a popular inference server that are necessary to bring the performance of a generic Triton attention kernel from 19.7% of the state-of-the-art to 105.9%. Our results highlight how open-source domain-specific languages can be leveraged to unlock model portability across different GPU vendors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_11581 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Anatomy of a Triton Attention Kernel Ringlein, Burkhard van Lunteren, Jan Stoica, Radu Parnell, Thomas Machine Learning Artificial Intelligence Computation and Language Distributed, Parallel, and Cluster Computing Programming Languages I.2; D.2; C.4; C.5 A long-standing goal in both industry and academia is to develop an LLM inference platform that is portable across hardware architectures, eliminates the need for low-level hand-tuning, and still delivers best-in-class efficiency. In this work, we demonstrate that portable, efficient cross-platform LLM inference is indeed possible and share our experience. We develop a state-of-the-art paged attention kernel, the core performance-critical component of many LLM deployments, that builds exclusively on the domain-specific just-in-time compiled language Triton to achieve state-of-the-art performance on both NVIDIA and AMD GPUs. We describe our high-level approach, the key algorithmic and system-level improvements, the parameter auto-tuning required to unlock efficiency, and the integrations into a popular inference server that are necessary to bring the performance of a generic Triton attention kernel from 19.7% of the state-of-the-art to 105.9%. Our results highlight how open-source domain-specific languages can be leveraged to unlock model portability across different GPU vendors. |
| title | The Anatomy of a Triton Attention Kernel |
| topic | Machine Learning Artificial Intelligence Computation and Language Distributed, Parallel, and Cluster Computing Programming Languages I.2; D.2; C.4; C.5 |
| url | https://arxiv.org/abs/2511.11581 |