Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2410.00273 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914961252417536 |
|---|---|
| author | Subramanian, Shashank Rrapaj, Ermal Harrington, Peter Chheda, Smeet Farrell, Steven Austin, Brian Williams, Samuel Wright, Nicholas Bhimji, Wahid |
| author_facet | Subramanian, Shashank Rrapaj, Ermal Harrington, Peter Chheda, Smeet Farrell, Steven Austin, Brian Williams, Samuel Wright, Nicholas Bhimji, Wahid |
| contents | Generative AI, in particular large transformer models, are increasingly driving HPC system design in science and industry. We analyze performance characteristics of such transformer models and discuss their sensitivity to the transformer type, parallelization strategy, and HPC system features (accelerators and interconnects). We utilize a performance model that allows us to explore this complex design space and highlight its key components. We find that different transformer types demand different parallelism and system characteristics at different training regimes. Large Language Models are performant with 3D parallelism and amplify network needs only at pre-training scales with reduced dependence on accelerator capacity and bandwidth. On the other hand, long-sequence transformers, representative of scientific foundation models, place a more uniform dependence on network and capacity with necessary 4D parallelism. Our analysis emphasizes the need for closer performance modeling of different transformer types keeping system features in mind and demonstrates a path towards this. Our code is available as open-source. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_00273 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Comprehensive Performance Modeling and System Design Insights for Foundation Models Subramanian, Shashank Rrapaj, Ermal Harrington, Peter Chheda, Smeet Farrell, Steven Austin, Brian Williams, Samuel Wright, Nicholas Bhimji, Wahid Machine Learning Distributed, Parallel, and Cluster Computing Generative AI, in particular large transformer models, are increasingly driving HPC system design in science and industry. We analyze performance characteristics of such transformer models and discuss their sensitivity to the transformer type, parallelization strategy, and HPC system features (accelerators and interconnects). We utilize a performance model that allows us to explore this complex design space and highlight its key components. We find that different transformer types demand different parallelism and system characteristics at different training regimes. Large Language Models are performant with 3D parallelism and amplify network needs only at pre-training scales with reduced dependence on accelerator capacity and bandwidth. On the other hand, long-sequence transformers, representative of scientific foundation models, place a more uniform dependence on network and capacity with necessary 4D parallelism. Our analysis emphasizes the need for closer performance modeling of different transformer types keeping system features in mind and demonstrates a path towards this. Our code is available as open-source. |
| title | Comprehensive Performance Modeling and System Design Insights for Foundation Models |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2410.00273 |