MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910167739662336 |
|---|---|
| author | Zhao, Juntao Lu, Qi Jia, Wei Wan, Borui Zuo, Lei Feng, Junda Jiang, Jianyu Chen, Yangrui Cao, Shuaishuai He, Jialing Jiang, Kaihua Hu, Yuanzhe Nong, Shibiao Peng, Yanghua Lin, Haibin Wu, Chuan |
| author_facet | Zhao, Juntao Lu, Qi Jia, Wei Wan, Borui Zuo, Lei Feng, Junda Jiang, Jianyu Chen, Yangrui Cao, Shuaishuai He, Jialing Jiang, Kaihua Hu, Yuanzhe Nong, Shibiao Peng, Yanghua Lin, Haibin Wu, Chuan |
| contents | Modern frameworks for training large foundation models (LFMs) employ dataloaders in a data-parallel manner, with each loader processing a disjoint subset of training data. When preparing data for LFM training that originates from multiple, distinct sources, two fundamental challenges arise. First, due to the quadratic computational complexity of the attention operator, the non-uniform sample distribution over data-parallel ranks leads to significant workload imbalance among dataloaders, degrading the training efficiency. Second, supporting diverse data sources requires per-dataset file access states that are redundantly replicated across parallel loaders, consuming excessive memory. This also hinders dynamic data mixing (e.g., curriculum learning) and causes redundant access/memory overhead in hybrid parallelism.
We present MegaScale-Data, an industrial-grade distributed data loading architecture for multisource LFMs training, with three key innovations: (1) Disaggregated data preprocessing via role-specific actors (Source Loaders/Data Constructors) to eliminate source and parallelism redundant data access and ensure multisource scalability. (2) Centralized and declarative data plane for load-time multisource orchestration, such as long-short context, multimodality, and curriculum learning. (3) Multi-level auto-partitioning and scaling mechanism for source loaders under heterogeneous preprocessing costs. We also contribute our designs and operational experience in deployment and fault tolerance. MegaScale-Data achieves up to: (1) 4.5x end-to-end training throughput improvement, and (2) 13.5x reduction in CPU memory usage. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09844 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training Zhao, Juntao Lu, Qi Jia, Wei Wan, Borui Zuo, Lei Feng, Junda Jiang, Jianyu Chen, Yangrui Cao, Shuaishuai He, Jialing Jiang, Kaihua Hu, Yuanzhe Nong, Shibiao Peng, Yanghua Lin, Haibin Wu, Chuan Distributed, Parallel, and Cluster Computing Artificial Intelligence Modern frameworks for training large foundation models (LFMs) employ dataloaders in a data-parallel manner, with each loader processing a disjoint subset of training data. When preparing data for LFM training that originates from multiple, distinct sources, two fundamental challenges arise. First, due to the quadratic computational complexity of the attention operator, the non-uniform sample distribution over data-parallel ranks leads to significant workload imbalance among dataloaders, degrading the training efficiency. Second, supporting diverse data sources requires per-dataset file access states that are redundantly replicated across parallel loaders, consuming excessive memory. This also hinders dynamic data mixing (e.g., curriculum learning) and causes redundant access/memory overhead in hybrid parallelism. We present MegaScale-Data, an industrial-grade distributed data loading architecture for multisource LFMs training, with three key innovations: (1) Disaggregated data preprocessing via role-specific actors (Source Loaders/Data Constructors) to eliminate source and parallelism redundant data access and ensure multisource scalability. (2) Centralized and declarative data plane for load-time multisource orchestration, such as long-short context, multimodality, and curriculum learning. (3) Multi-level auto-partitioning and scaling mechanism for source loaders under heterogeneous preprocessing costs. We also contribute our designs and operational experience in deployment and fault tolerance. MegaScale-Data achieves up to: (1) 4.5x end-to-end training throughput improvement, and (2) 13.5x reduction in CPU memory usage. |
| title | MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2504.09844 |