Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916481418133504 |
|---|---|
| author | De Sensi, Daniele Pichetti, Lorenzo Vella, Flavio De Matteis, Tiziano Ren, Zebin Fusco, Luigi Turisini, Matteo Cesarini, Daniele Lust, Kurt Trivedi, Animesh Roweth, Duncan Spiga, Filippo Di Girolamo, Salvatore Hoefler, Torsten |
| author_facet | De Sensi, Daniele Pichetti, Lorenzo Vella, Flavio De Matteis, Tiziano Ren, Zebin Fusco, Luigi Turisini, Matteo Cesarini, Daniele Lust, Kurt Trivedi, Animesh Roweth, Duncan Spiga, Filippo Di Girolamo, Salvatore Hoefler, Torsten |
| contents | Multi-GPU nodes are increasingly common in the rapidly evolving landscape of exascale supercomputers. On these systems, GPUs on the same node are connected through dedicated networks, with bandwidths up to a few terabits per second. However, gauging performance expectations and maximizing system efficiency is challenging due to different technologies, design options, and software layers. This paper comprehensively characterizes three supercomputers - Alps, Leonardo, and LUMI - each with a unique architecture and design. We focus on performance evaluation of intra-node and inter-node interconnects on up to 4096 GPUs, using a mix of intra-node and inter-node benchmarks. By analyzing its limitations and opportunities, we aim to offer practical guidance to researchers, system architects, and software developers dealing with multi-GPU supercomputing. Our results show that there is untapped bandwidth, and there are still many opportunities for optimization, ranging from network to software optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_14090 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects De Sensi, Daniele Pichetti, Lorenzo Vella, Flavio De Matteis, Tiziano Ren, Zebin Fusco, Luigi Turisini, Matteo Cesarini, Daniele Lust, Kurt Trivedi, Animesh Roweth, Duncan Spiga, Filippo Di Girolamo, Salvatore Hoefler, Torsten Distributed, Parallel, and Cluster Computing Artificial Intelligence Hardware Architecture Networking and Internet Architecture Performance C.2.4; C.5.1; C.2.1; C.4 Multi-GPU nodes are increasingly common in the rapidly evolving landscape of exascale supercomputers. On these systems, GPUs on the same node are connected through dedicated networks, with bandwidths up to a few terabits per second. However, gauging performance expectations and maximizing system efficiency is challenging due to different technologies, design options, and software layers. This paper comprehensively characterizes three supercomputers - Alps, Leonardo, and LUMI - each with a unique architecture and design. We focus on performance evaluation of intra-node and inter-node interconnects on up to 4096 GPUs, using a mix of intra-node and inter-node benchmarks. By analyzing its limitations and opportunities, we aim to offer practical guidance to researchers, system architects, and software developers dealing with multi-GPU supercomputing. Our results show that there is untapped bandwidth, and there are still many opportunities for optimization, ranging from network to software optimization. |
| title | Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Hardware Architecture Networking and Internet Architecture Performance C.2.4; C.5.1; C.2.1; C.4 |
| url | https://arxiv.org/abs/2408.14090 |