Accelerating Compound LLM Training Workloads with Maestro
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913112605589504 |
|---|---|
| author | Yuan, Xiulong Chen, Hongqing Peng, Jiaxuan Zhou, Fan Ruan, Zhixiang Wang, Zekun Zheng, Bo Men, Rui Wang, Haiquan Zhang, Zhipeng Chen, Langshi Yuan, Man Gao, Jiaqi Qian, Zhengping Lin, Junyang Li, Yong Lin, Wei Wang, Junhua Zhou, Jingren |
| author_facet | Yuan, Xiulong Chen, Hongqing Peng, Jiaxuan Zhou, Fan Ruan, Zhixiang Wang, Zekun Zheng, Bo Men, Rui Wang, Haiquan Zhang, Zhipeng Chen, Langshi Yuan, Man Gao, Jiaqi Qian, Zhengping Lin, Junyang Li, Yong Lin, Wei Wang, Junhua Zhou, Jingren |
| contents | Compound LLM training workloads-such as knowledge distillation and multimodal LLM (MLLM) training-are gaining prominence. These typically comprise heterogeneous components differing in parameter scale, execution mode (forward-only or full forward-backward), and sequence length. Besides, component activation can be data-dependent: in MLLM training, modality-specific parts activate only when inputs contain corresponding modalities, causing dynamic computational paths and irregular runtime workloads. Conventional frameworks, designed for monolithic models, cannot handle the dual heterogeneity-static (across components) and dynamic (runtime). By enforcing one-size-fits-all training configurations across components and ignoring input-induced variations, they suffer suboptimal throughput and poor GPU utilization. In this paper, we introduce Maestro, a section-centric training framework that addresses both challenges. Maestro first restructures the workload into a coarse-grained section graph. Each section independently configures its parallelism strategy, micro-batch size, and data-parallel degree-enabling fine-grained, component-aware resource allocation to tackle static heterogeneity. To tackle runtime irregularity, Maestro introduces a wavefront scheduling algorithm that dynamically reorders input samples to orchestrate concurrent section execution while preserving cross-section data dependencies. This maximizes inter-section parallelism and minimizes stalls, boosting hardware utilization. Deployed in production for millions of GPU hours, Maestro reduces GPU consumption by ~40% on key workloads-including knowledge distillation and MLLM training-validating its real-world impact. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_10501 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Accelerating Compound LLM Training Workloads with Maestro Yuan, Xiulong Chen, Hongqing Peng, Jiaxuan Zhou, Fan Ruan, Zhixiang Wang, Zekun Zheng, Bo Men, Rui Wang, Haiquan Zhang, Zhipeng Chen, Langshi Yuan, Man Gao, Jiaqi Qian, Zhengping Lin, Junyang Li, Yong Lin, Wei Wang, Junhua Zhou, Jingren Distributed, Parallel, and Cluster Computing Compound LLM training workloads-such as knowledge distillation and multimodal LLM (MLLM) training-are gaining prominence. These typically comprise heterogeneous components differing in parameter scale, execution mode (forward-only or full forward-backward), and sequence length. Besides, component activation can be data-dependent: in MLLM training, modality-specific parts activate only when inputs contain corresponding modalities, causing dynamic computational paths and irregular runtime workloads. Conventional frameworks, designed for monolithic models, cannot handle the dual heterogeneity-static (across components) and dynamic (runtime). By enforcing one-size-fits-all training configurations across components and ignoring input-induced variations, they suffer suboptimal throughput and poor GPU utilization. In this paper, we introduce Maestro, a section-centric training framework that addresses both challenges. Maestro first restructures the workload into a coarse-grained section graph. Each section independently configures its parallelism strategy, micro-batch size, and data-parallel degree-enabling fine-grained, component-aware resource allocation to tackle static heterogeneity. To tackle runtime irregularity, Maestro introduces a wavefront scheduling algorithm that dynamically reorders input samples to orchestrate concurrent section execution while preserving cross-section data dependencies. This maximizes inter-section parallelism and minimizes stalls, boosting hardware utilization. Deployed in production for millions of GPU hours, Maestro reduces GPU consumption by ~40% on key workloads-including knowledge distillation and MLLM training-validating its real-world impact. |
| title | Accelerating Compound LLM Training Workloads with Maestro |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2605.10501 |