MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911667323928576 |
|---|---|
| author | Xue, Chunyu Chen, Yangrui Jiang, Jianyu Zheng, Ningxin Feng, Junda Chen, Jingji Zhao, Shixiong Yan, Shen Lin, Yi Shi, Lei Wang, Zanbo Luo, Lishu Wu, Faming Lin, Haibin Liu, Xin Peng, Yanghua Chen, Quan |
| author_facet | Xue, Chunyu Chen, Yangrui Jiang, Jianyu Zheng, Ningxin Feng, Junda Chen, Jingji Zhao, Shixiong Yan, Shen Lin, Yi Shi, Lei Wang, Zanbo Luo, Lishu Wu, Faming Lin, Haibin Liu, Xin Peng, Yanghua Chen, Quan |
| contents | As the foundational component of versatile AI applications, training an multimodal large language model (MLLM) relies on multimodal datasets with dynamic modality mixture proportions and sample length distributions. However, existing MLLM systems remain inefficient under dynamic workloads, due to statically coupled decisions of resource allocation and model parallelization between encoders and the LLM backbone. This paper presents MegaScale-Omni, an industrial-grade MLLM training system tailored for dynamic workload adaption and hyper-scale deployment. MegaScale-Omni is built upon the training scheme of encoder-LLM multiplexing with three key innovations: (1) Decoupled parallelism strategies with long-short sequence parallelism for encoders to process variable-length samples, and full-fledged 5D parallelism for the LLM backbone, both organized under a communication-efficient parallelization layout. (2) Unified encoder-LLM representations for flexible, extensible colocation, and a new paradigm of encoder-LLM joint pipeline with workload resilience. (3) Workload balancing techniques via decentralized grouped reordering in data loaders and adaptive resharding from encoder to LLM ranks. MegaScale-Omni is deployed as the foundation of our in-house large-scale MLLM training tasks with thousands of GPUs. Our experimental results demonstrate $1.27\times$-$7.57\times$ throughput improvement under production-grade dynamic workloads, as compared to four state-of-the-art systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_08962 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production Xue, Chunyu Chen, Yangrui Jiang, Jianyu Zheng, Ningxin Feng, Junda Chen, Jingji Zhao, Shixiong Yan, Shen Lin, Yi Shi, Lei Wang, Zanbo Luo, Lishu Wu, Faming Lin, Haibin Liu, Xin Peng, Yanghua Chen, Quan Distributed, Parallel, and Cluster Computing As the foundational component of versatile AI applications, training an multimodal large language model (MLLM) relies on multimodal datasets with dynamic modality mixture proportions and sample length distributions. However, existing MLLM systems remain inefficient under dynamic workloads, due to statically coupled decisions of resource allocation and model parallelization between encoders and the LLM backbone. This paper presents MegaScale-Omni, an industrial-grade MLLM training system tailored for dynamic workload adaption and hyper-scale deployment. MegaScale-Omni is built upon the training scheme of encoder-LLM multiplexing with three key innovations: (1) Decoupled parallelism strategies with long-short sequence parallelism for encoders to process variable-length samples, and full-fledged 5D parallelism for the LLM backbone, both organized under a communication-efficient parallelization layout. (2) Unified encoder-LLM representations for flexible, extensible colocation, and a new paradigm of encoder-LLM joint pipeline with workload resilience. (3) Workload balancing techniques via decentralized grouped reordering in data loaders and adaptive resharding from encoder to LLM ranks. MegaScale-Omni is deployed as the foundation of our in-house large-scale MLLM training tasks with thousands of GPUs. Our experimental results demonstrate $1.27\times$-$7.57\times$ throughput improvement under production-grade dynamic workloads, as compared to four state-of-the-art systems. |
| title | MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2605.08962 |