Serving Large Language Models on Huawei CloudMatrix384
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918065378164736 |
|---|---|
| author | Zuo, Pengfei Lin, Huimin Deng, Junbo Zou, Nan Yang, Xingkun Diao, Yingyu Gao, Weifeng Xu, Ke Chen, Zhangyu Lu, Shirui Qiu, Zhao Li, Peiyang Chang, Xianyu Yu, Zhengzhong Miao, Fangzheng Zheng, Jia Li, Ying Feng, Yuan Wang, Bei Zong, Zaijian Zhou, Mosong Zhou, Wenli Chen, Houjiang Liao, Xingyu Li, Yipeng Zhang, Wenxiao Zhu, Ping Wang, Yinggang Xiao, Chuanjie Liang, Depeng Cao, Dong Liu, Juncheng Yang, Yongqiang Bai, Xiaolong Li, Yi Xie, Huaguo Wu, Huatao Yu, Zhibin Chen, Lv Liu, Hu Ding, Yujun Zhu, Haipei Xia, Jing Xiong, Yi Yu, Zhou Liao, Heng |
| author_facet | Zuo, Pengfei Lin, Huimin Deng, Junbo Zou, Nan Yang, Xingkun Diao, Yingyu Gao, Weifeng Xu, Ke Chen, Zhangyu Lu, Shirui Qiu, Zhao Li, Peiyang Chang, Xianyu Yu, Zhengzhong Miao, Fangzheng Zheng, Jia Li, Ying Feng, Yuan Wang, Bei Zong, Zaijian Zhou, Mosong Zhou, Wenli Chen, Houjiang Liao, Xingyu Li, Yipeng Zhang, Wenxiao Zhu, Ping Wang, Yinggang Xiao, Chuanjie Liang, Depeng Cao, Dong Liu, Juncheng Yang, Yongqiang Bai, Xiaolong Li, Yi Xie, Huaguo Wu, Huatao Yu, Zhibin Chen, Lv Liu, Hu Ding, Yujun Zhu, Haipei Xia, Jing Xiong, Yi Yu, Zhou Liao, Heng |
| contents | The rapid evolution of large language models (LLMs), driven by growing parameter scales, adoption of mixture-of-experts (MoE) architectures, and expanding context lengths, imposes unprecedented demands on AI infrastructure. Traditional AI clusters face limitations in compute intensity, memory bandwidth, inter-chip communication, and latency, compounded by variable workloads and strict service-level objectives. Addressing these issues requires fundamentally redesigned hardware-software integration. This paper introduces Huawei CloudMatrix, a next-generation AI datacenter architecture, realized in the production-grade CloudMatrix384 supernode. It integrates 384 Ascend 910 NPUs and 192 Kunpeng CPUs interconnected via an ultra-high-bandwidth Unified Bus (UB) network, enabling direct all-to-all communication and dynamic pooling of resources. These features optimize performance for communication-intensive operations, such as large-scale MoE expert parallelism and distributed key-value cache access. To fully leverage CloudMatrix384, we propose CloudMatrix-Infer, an advanced LLM serving solution incorporating three core innovations: a peer-to-peer serving architecture that independently scales prefill, decode, and caching; a large-scale expert parallelism strategy supporting EP320 via efficient UB-based token dispatch; and hardware-aware optimizations including specialized operators, microbatch-based pipelining, and INT8 quantization. Evaluation with the DeepSeek-R1 model shows CloudMatrix-Infer achieves state-of-the-art efficiency: prefill throughput of 6,688 tokens/s per NPU and decode throughput of 1,943 tokens/s per NPU (<50 ms TPOT). It effectively balances throughput and latency, sustaining 538 tokens/s per NPU even under stringent 15 ms latency constraints, while INT8 quantization maintains model accuracy across benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_12708 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Serving Large Language Models on Huawei CloudMatrix384 Zuo, Pengfei Lin, Huimin Deng, Junbo Zou, Nan Yang, Xingkun Diao, Yingyu Gao, Weifeng Xu, Ke Chen, Zhangyu Lu, Shirui Qiu, Zhao Li, Peiyang Chang, Xianyu Yu, Zhengzhong Miao, Fangzheng Zheng, Jia Li, Ying Feng, Yuan Wang, Bei Zong, Zaijian Zhou, Mosong Zhou, Wenli Chen, Houjiang Liao, Xingyu Li, Yipeng Zhang, Wenxiao Zhu, Ping Wang, Yinggang Xiao, Chuanjie Liang, Depeng Cao, Dong Liu, Juncheng Yang, Yongqiang Bai, Xiaolong Li, Yi Xie, Huaguo Wu, Huatao Yu, Zhibin Chen, Lv Liu, Hu Ding, Yujun Zhu, Haipei Xia, Jing Xiong, Yi Yu, Zhou Liao, Heng Distributed, Parallel, and Cluster Computing Artificial Intelligence Hardware Architecture Machine Learning The rapid evolution of large language models (LLMs), driven by growing parameter scales, adoption of mixture-of-experts (MoE) architectures, and expanding context lengths, imposes unprecedented demands on AI infrastructure. Traditional AI clusters face limitations in compute intensity, memory bandwidth, inter-chip communication, and latency, compounded by variable workloads and strict service-level objectives. Addressing these issues requires fundamentally redesigned hardware-software integration. This paper introduces Huawei CloudMatrix, a next-generation AI datacenter architecture, realized in the production-grade CloudMatrix384 supernode. It integrates 384 Ascend 910 NPUs and 192 Kunpeng CPUs interconnected via an ultra-high-bandwidth Unified Bus (UB) network, enabling direct all-to-all communication and dynamic pooling of resources. These features optimize performance for communication-intensive operations, such as large-scale MoE expert parallelism and distributed key-value cache access. To fully leverage CloudMatrix384, we propose CloudMatrix-Infer, an advanced LLM serving solution incorporating three core innovations: a peer-to-peer serving architecture that independently scales prefill, decode, and caching; a large-scale expert parallelism strategy supporting EP320 via efficient UB-based token dispatch; and hardware-aware optimizations including specialized operators, microbatch-based pipelining, and INT8 quantization. Evaluation with the DeepSeek-R1 model shows CloudMatrix-Infer achieves state-of-the-art efficiency: prefill throughput of 6,688 tokens/s per NPU and decode throughput of 1,943 tokens/s per NPU (<50 ms TPOT). It effectively balances throughput and latency, sustaining 538 tokens/s per NPU even under stringent 15 ms latency constraints, while INT8 quantization maintains model accuracy across benchmarks. |
| title | Serving Large Language Models on Huawei CloudMatrix384 |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Hardware Architecture Machine Learning |
| url | https://arxiv.org/abs/2506.12708 |