TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912520122400768 |
|---|---|
| author | Shi, Ziji Jiang, Le Wang, Ang Zhang, Jie Wu, Chencan Li, Yong Xiao, Xiaokui Lin, Wei Li, Jialin |
| author_facet | Shi, Ziji Jiang, Le Wang, Ang Zhang, Jie Wu, Chencan Li, Yong Xiao, Xiaokui Lin, Wei Li, Jialin |
| contents | Tensor parallelism is an essential technique for distributed training of large neural networks. However, automatically determining an optimal tensor parallel strategy is challenging due to the gigantic search space, which grows exponentially with model size and tensor dimension. This prohibits the adoption of auto-parallel systems on larger models.
We observe that neural networks usually contain repeated substructures, and build an automatic parallelism framework named TAPAS that eliminates redundant search efforts. TAPAS employs a divide-and-conquer approach that efficiently folds the search space by identifying those unique substructures. As a result, it runs at sub-linear complexity concerning the model size, making it a scalable solution for training large-scale networks. Our evaluations demonstrate that TAPAS outperforms the state-of-the-art automatic parallelism frameworks by up to $160\times$ in search speed on a wide range of models, and the performance of derived strategies is competitive or even better compared with the expert-engineered Megatron-LM library. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_00247 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networks Shi, Ziji Jiang, Le Wang, Ang Zhang, Jie Wu, Chencan Li, Yong Xiao, Xiaokui Lin, Wei Li, Jialin Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Tensor parallelism is an essential technique for distributed training of large neural networks. However, automatically determining an optimal tensor parallel strategy is challenging due to the gigantic search space, which grows exponentially with model size and tensor dimension. This prohibits the adoption of auto-parallel systems on larger models. We observe that neural networks usually contain repeated substructures, and build an automatic parallelism framework named TAPAS that eliminates redundant search efforts. TAPAS employs a divide-and-conquer approach that efficiently folds the search space by identifying those unique substructures. As a result, it runs at sub-linear complexity concerning the model size, making it a scalable solution for training large-scale networks. Our evaluations demonstrate that TAPAS outperforms the state-of-the-art automatic parallelism frameworks by up to $160\times$ in search speed on a wide range of models, and the performance of derived strategies is competitive or even better compared with the expert-engineered Megatron-LM library. |
| title | TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networks |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2302.00247 |