Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.14158 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913487776645120 |
|---|---|
| author | An, Wei Bi, Xiao Chen, Guanting Chen, Shanhuang Deng, Chengqi Ding, Honghui Dong, Kai Du, Qiushi Gao, Wenjun Guan, Kang Guo, Jianzhong Guo, Yongqiang Fu, Zhe He, Ying Huang, Panpan Li, Jiashi Liang, Wenfeng Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Yuxuan Lu, Shanghao Lu, Xuan Nie, Xiaotao Pei, Tian Qiu, Junjie Qu, Hui Ren, Zehui Sha, Zhangli Su, Xuecheng Sun, Xiaowen Tan, Yixuan Tang, Minghui Wang, Shiyu Wang, Yaohui Wang, Yongji Xie, Ziwei Xiong, Yiliang Xu, Yanhong Ye, Shengfeng Yu, Shuiping Zha, Yukun Zhang, Liyue Zhang, Haowei Zhang, Mingchuan Zhang, Wentao Zhang, Yichao Zhao, Chenggang Zhao, Yao Zhou, Shangyan Zhou, Shunfeng Zou, Yuheng |
| author_facet | An, Wei Bi, Xiao Chen, Guanting Chen, Shanhuang Deng, Chengqi Ding, Honghui Dong, Kai Du, Qiushi Gao, Wenjun Guan, Kang Guo, Jianzhong Guo, Yongqiang Fu, Zhe He, Ying Huang, Panpan Li, Jiashi Liang, Wenfeng Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Yuxuan Lu, Shanghao Lu, Xuan Nie, Xiaotao Pei, Tian Qiu, Junjie Qu, Hui Ren, Zehui Sha, Zhangli Su, Xuecheng Sun, Xiaowen Tan, Yixuan Tang, Minghui Wang, Shiyu Wang, Yaohui Wang, Yongji Xie, Ziwei Xiong, Yiliang Xu, Yanhong Ye, Shengfeng Yu, Shuiping Zha, Yukun Zhang, Liyue Zhang, Haowei Zhang, Mingchuan Zhang, Wentao Zhang, Yichao Zhao, Chenggang Zhao, Yao Zhou, Shangyan Zhou, Shunfeng Zou, Yuheng |
| contents | The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_14158 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning An, Wei Bi, Xiao Chen, Guanting Chen, Shanhuang Deng, Chengqi Ding, Honghui Dong, Kai Du, Qiushi Gao, Wenjun Guan, Kang Guo, Jianzhong Guo, Yongqiang Fu, Zhe He, Ying Huang, Panpan Li, Jiashi Liang, Wenfeng Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Yuxuan Lu, Shanghao Lu, Xuan Nie, Xiaotao Pei, Tian Qiu, Junjie Qu, Hui Ren, Zehui Sha, Zhangli Su, Xuecheng Sun, Xiaowen Tan, Yixuan Tang, Minghui Wang, Shiyu Wang, Yaohui Wang, Yongji Xie, Ziwei Xiong, Yiliang Xu, Yanhong Ye, Shengfeng Yu, Shuiping Zha, Yukun Zhang, Liyue Zhang, Haowei Zhang, Mingchuan Zhang, Wentao Zhang, Yichao Zhao, Chenggang Zhao, Yao Zhou, Shangyan Zhou, Shunfeng Zou, Yuheng Distributed, Parallel, and Cluster Computing Artificial Intelligence The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC. |
| title | Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2408.14158 |