AttentionEngine: A Versatile Framework for Efficient Attention Mechanisms on Diverse Hardware Platforms
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910838251585536 |
|---|---|
| author | Chen, Feiyang Cheng, Yu Wang, Lei Xia, Yuqing Miao, Ziming Ma, Lingxiao Yang, Fan Xue, Jilong Yang, Zhi Yang, Mao Chen, Haibo |
| author_facet | Chen, Feiyang Cheng, Yu Wang, Lei Xia, Yuqing Miao, Ziming Ma, Lingxiao Yang, Fan Xue, Jilong Yang, Zhi Yang, Mao Chen, Haibo |
| contents | Transformers and large language models (LLMs) have revolutionized machine learning, with attention mechanisms at the core of their success. As the landscape of attention variants expands, so too do the challenges of optimizing their performance, particularly across different hardware platforms. Current optimization strategies are often narrowly focused, requiring extensive manual intervention to accommodate changes in model configurations or hardware environments. In this paper, we introduce AttentionEngine, a comprehensive framework designed to streamline the optimization of attention mechanisms across heterogeneous hardware backends. By decomposing attention computation into modular operations with customizable components, AttentionEngine enables flexible adaptation to diverse algorithmic requirements. The framework further automates kernel optimization through a combination of programmable templates and a robust cross-platform scheduling strategy. Empirical results reveal performance gains of up to 10x on configurations beyond the reach of existing methods. AttentionEngine offers a scalable, efficient foundation for developing and deploying attention mechanisms with minimal manual tuning. Our code has been open-sourced and is available at https://github.com/microsoft/AttentionEngine. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15349 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AttentionEngine: A Versatile Framework for Efficient Attention Mechanisms on Diverse Hardware Platforms Chen, Feiyang Cheng, Yu Wang, Lei Xia, Yuqing Miao, Ziming Ma, Lingxiao Yang, Fan Xue, Jilong Yang, Zhi Yang, Mao Chen, Haibo Computation and Language Machine Learning Performance Transformers and large language models (LLMs) have revolutionized machine learning, with attention mechanisms at the core of their success. As the landscape of attention variants expands, so too do the challenges of optimizing their performance, particularly across different hardware platforms. Current optimization strategies are often narrowly focused, requiring extensive manual intervention to accommodate changes in model configurations or hardware environments. In this paper, we introduce AttentionEngine, a comprehensive framework designed to streamline the optimization of attention mechanisms across heterogeneous hardware backends. By decomposing attention computation into modular operations with customizable components, AttentionEngine enables flexible adaptation to diverse algorithmic requirements. The framework further automates kernel optimization through a combination of programmable templates and a robust cross-platform scheduling strategy. Empirical results reveal performance gains of up to 10x on configurations beyond the reach of existing methods. AttentionEngine offers a scalable, efficient foundation for developing and deploying attention mechanisms with minimal manual tuning. Our code has been open-sourced and is available at https://github.com/microsoft/AttentionEngine. |
| title | AttentionEngine: A Versatile Framework for Efficient Attention Mechanisms on Diverse Hardware Platforms |
| topic | Computation and Language Machine Learning Performance |
| url | https://arxiv.org/abs/2502.15349 |