Image de couverture de livre

MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Xue, Chunyu, Chen, Yangrui, Jiang, Jianyu, Zheng, Ningxin, Feng, Junda, Chen, Jingji, Zhao, Shixiong, Yan, Shen, Lin, Yi, Shi, Lei, Wang, Zanbo, Luo, Lishu, Wu, Faming, Lin, Haibin, Liu, Xin, Peng, Yanghua, Chen, Quan
Format:	Preprint
Publié:	2026
Sujets:	Distributed, Parallel, and Cluster Computing
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training
par: Zhao, Juntao, et autres
Publié: (2025)

MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
par: Jin, Chao, et autres
Publié: (2025)

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
par: Jiang, Ziheng, et autres
Publié: (2024)

MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
par: Zhu, Ruidong, et autres
Publié: (2025)

Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
par: Feng, Weiqi, et autres
Publié: (2024)

Mesh-Attention: A New Communication-Efficient Distributed Attention with Improved Data Locality
par: Chen, Sirui, et autres
Publié: (2025)

VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo
par: Ma, Qianli, et autres
Publié: (2025)

MegaFlow: Large-Scale Distributed Orchestration System for the Agentic Era
par: Zhang, Lei, et autres
Publié: (2026)

ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs
par: Ge, Hao, et autres
Publié: (2025)

Large-Scale LLM Inference with Heterogeneous Workloads: Prefill-Decode Contention and Asymptotically Optimal Control
par: Lin, Ruihan, et autres
Publié: (2026)

LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
par: Zhao, Juntao, et autres
Publié: (2024)

veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD
par: Li, Youjie, et autres
Publié: (2025)

Efficient Unified Caching for Accelerating Heterogeneous AI Workloads
par: Wang, Tianze, et autres
Publié: (2025)

Fast-HotStuff: A Fast and Resilient HotStuff Protocol
par: Jalalzai, Mohammad M., et autres
Publié: (2020)

QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices
par: Zhao, Juntao, et autres
Publié: (2024)

Profiling and Modeling of Power Characteristics of Leadership-Scale HPC System Workloads
par: Karimi, Ahmad Maroof, et autres
Publié: (2024)

Accelerating Compound LLM Training Workloads with Maestro
par: Yuan, Xiulong, et autres
Publié: (2026)

Scheduling Data-Intensive Workloads in Large-Scale Distributed Systems: Trends and Challenges
par: Stavrinides, Georgios L., et autres
Publié: (2025)

PRISM: Dynamic Primitive-Based Forecasting for Large-Scale GPU Cluster Workloads
par: Wu, Xin, et autres
Publié: (2026)

LLMSched: Uncertainty-Aware Workload Scheduling for Compound LLM Applications
par: Zhu, Botao, et autres
Publié: (2025)

Beyond Microservices: Testing Web-Scale RCA Methods on GPU-Driven LLM Workloads
par: Scheinert, Dominik, et autres
Publié: (2026)

veScale-FSDP: Flexible and High-Performance FSDP at Scale
par: Wang, Zezhou, et autres
Publié: (2026)

City-Scale Visibility Graph Analysis via GPU-Accelerated HyperBall
par: Hodge, Alex, et autres
Publié: (2026)

Calibre: Towards Fair and Accurate Personalized Federated Learning with Self-Supervised Learning
par: Chen, Sijia, et autres
Publié: (2024)

Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts
par: Zhang, Shulai, et autres
Publié: (2025)

Hurry: Dynamic Collaborative Framework For Low-orbit Mega-Constellation Data Downloading
par: Luo, Handong, et autres
Publié: (2024)

Hydra: Brokering Cloud and HPC Resources to Support the Execution of Heterogeneous Workloads at Scale
par: Alsaadi, Aymen, et autres
Publié: (2024)

TierBase: A Workload-Driven Cost-Optimized Key-Value Store
par: Shen, Zhitao, et autres
Publié: (2025)

Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models
par: Chen, Daoyuan, et autres
Publié: (2024)

TempoScale: A Cloud Workloads Prediction Approach Integrating Short-Term and Long-Term Information
par: Wen, Linfeng, et autres
Publié: (2024)

Mosaic: Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing
par: Wang, Yanbo, et autres
Publié: (2026)

SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding
par: Zhang, Ziyi, et autres
Publié: (2025)

WRATH: Workload Resilience Across Task Hierarchies in Task-based Parallel Programming Frameworks
par: Zhou, Sicheng, et autres
Publié: (2025)

Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Survey
par: Liang, Feng, et autres
Publié: (2024)

Resilient Auto-Scaling of Microservice Architectures with Efficient Resource Management
par: Ahmad, Hussain, et autres
Publié: (2025)

Crossword: Adaptive Consensus for Dynamic Data-Heavy Workloads
par: Hu, Guanzhou, et autres
Publié: (2025)

Making MoE-based LLM Inference Resilient with Tarragon
par: Zhang, Songyu, et autres
Publié: (2026)

Accelerating Gaussian beam tracing method with dynamic parallelism on graphics processing units
par: Sheng, Zhang, et autres
Publié: (2025)

A Tale of Two Scales: Reconciling Horizontal and Vertical Scaling for Inference Serving Systems
par: Razavi, Kamran, et autres
Publié: (2024)

SWARM+: Scalable and Resilient Multi-Agent Consensus for Fully-Decentralized Data-Aware Workload Management
par: Thareja, Komal, et autres
Publié: (2026)