The Case for Co-Designing Model Architectures with Hardware
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913216612794368 |
|---|---|
| author | Anthony, Quentin Hatef, Jacob Narayanan, Deepak Biderman, Stella Bekman, Stas Yin, Junqi Shafi, Aamir Subramoni, Hari Panda, Dhabaleswar |
| author_facet | Anthony, Quentin Hatef, Jacob Narayanan, Deepak Biderman, Stella Bekman, Stas Yin, Junqi Shafi, Aamir Subramoni, Hari Panda, Dhabaleswar |
| contents | While GPUs are responsible for training the vast majority of state-of-the-art deep learning models, the implications of their architecture are often overlooked when designing new deep learning (DL) models. As a consequence, modifying a DL model to be more amenable to the target hardware can significantly improve the runtime performance of DL training and inference. In this paper, we provide a set of guidelines for users to maximize the runtime performance of their transformer models. These guidelines have been created by carefully considering the impact of various model hyperparameters controlling model shape on the efficiency of the underlying computation kernels executed on the GPU. We find the throughput of models with efficient model shapes is up to 39\% higher while preserving accuracy compared to models with a similar number of parameters but with unoptimized shapes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_14489 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Case for Co-Designing Model Architectures with Hardware Anthony, Quentin Hatef, Jacob Narayanan, Deepak Biderman, Stella Bekman, Stas Yin, Junqi Shafi, Aamir Subramoni, Hari Panda, Dhabaleswar Distributed, Parallel, and Cluster Computing Artificial Intelligence While GPUs are responsible for training the vast majority of state-of-the-art deep learning models, the implications of their architecture are often overlooked when designing new deep learning (DL) models. As a consequence, modifying a DL model to be more amenable to the target hardware can significantly improve the runtime performance of DL training and inference. In this paper, we provide a set of guidelines for users to maximize the runtime performance of their transformer models. These guidelines have been created by carefully considering the impact of various model hyperparameters controlling model shape on the efficiency of the underlying computation kernels executed on the GPU. We find the throughput of models with efficient model shapes is up to 39\% higher while preserving accuracy compared to models with a similar number of parameters but with unoptimized shapes. |
| title | The Case for Co-Designing Model Architectures with Hardware |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2401.14489 |