Optimizing Layer-Fused Scheduling of Transformer Networks on Multi-accelerator Platforms
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917694012391424 |
|---|---|
| author | Colleman, Steven Symons, Arne Jung, Victor J. B. Verhelst, Marian |
| author_facet | Colleman, Steven Symons, Arne Jung, Victor J. B. Verhelst, Marian |
| contents | The impact of transformer networks is booming, yet, they come with significant computational complexity. It is therefore essential to understand how to optimally map and execute these networks on modern neural processor hardware. So far, literature on transformer scheduling optimization has been focusing on deployment on GPU and specific ASICs. This work enables extensive hardware/mapping exploration by extending the DSE framework Stream towards support for transformers across a wide variety of hardware architectures and different execution schedules. After validation, we explore the optimal schedule for transformer layers/attention heads and investigate whether layer fusion is beneficial to improve latency, energy or memory requirements. Our study shows that the memory requirements for active feature data can be drastically reduced, by adapting the execution schedule based on the size of the input of the attention head. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09804 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Optimizing Layer-Fused Scheduling of Transformer Networks on Multi-accelerator Platforms Colleman, Steven Symons, Arne Jung, Victor J. B. Verhelst, Marian Hardware Architecture The impact of transformer networks is booming, yet, they come with significant computational complexity. It is therefore essential to understand how to optimally map and execute these networks on modern neural processor hardware. So far, literature on transformer scheduling optimization has been focusing on deployment on GPU and specific ASICs. This work enables extensive hardware/mapping exploration by extending the DSE framework Stream towards support for transformers across a wide variety of hardware architectures and different execution schedules. After validation, we explore the optimal schedule for transformer layers/attention heads and investigate whether layer fusion is beneficial to improve latency, energy or memory requirements. Our study shows that the memory requirements for active feature data can be drastically reduced, by adapting the execution schedule based on the size of the input of the attention head. |
| title | Optimizing Layer-Fused Scheduling of Transformer Networks on Multi-accelerator Platforms |
| topic | Hardware Architecture |
| url | https://arxiv.org/abs/2406.09804 |