Copertina

Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Li, Haoyang, Fu, Fangcheng, Ge, Hao, Lin, Sheng, Wang, Xuanyu, Niu, Jiawen, Wang, Yujie, Zhang, Hailin, Nie, Xiaonan, Cui, Bin
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Distributed, Parallel, and Cluster Computing
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment
di: Li, Haoyang, et al.
Pubblicazione: (2024)

Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation
di: Wu, Tianyuan, et al.
Pubblicazione: (2025)

Improving Automatic Parallel Training via Balanced Memory Workload Optimization
di: Wang, Yujie, et al.
Pubblicazione: (2023)

LobRA: Multi-tenant Fine-tuning over Heterogeneous Data
di: Lin, Sheng, et al.
Pubblicazione: (2025)

Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations
di: Li, Haoyang, et al.
Pubblicazione: (2025)

FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Training
di: Wu, Tianyuan, et al.
Pubblicazione: (2024)

HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware
di: Yan, Ran, et al.
Pubblicazione: (2024)

FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
di: Wang, Yujie, et al.
Pubblicazione: (2024)

InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training
di: Wang, Shiju, et al.
Pubblicazione: (2025)

Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP
di: Zhao, Yilong, et al.
Pubblicazione: (2026)

Straggler Tolerant and Resilient DL Training on Homogeneous GPUs
di: Zhang, Zeyu, et al.
Pubblicazione: (2025)

Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models
di: Wang, Wei, et al.
Pubblicazione: (2024)

Unleashing Efficient Asynchronous RL Post-Training via Staleness-Constrained Rollout Coordination
di: Li, Haoyang, et al.
Pubblicazione: (2026)

LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing
di: Nie, Xiaonan, et al.
Pubblicazione: (2024)

MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training
di: Zhao, Pinxue, et al.
Pubblicazione: (2024)

DHP: Efficient Scaling of MLLM Training with Dynamic Hybrid Parallelism
di: Niu, Yifan, et al.
Pubblicazione: (2026)

Galvatron: An Automatic Distributed System for Efficient Foundation Model Training
di: Liu, Xinyi, et al.
Pubblicazione: (2025)

ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism
di: Ma, Tenghui, et al.
Pubblicazione: (2026)

ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training
di: Kang, Xueze, et al.
Pubblicazione: (2025)

Understanding Stragglers in Large Model Training Using What-if Analysis
di: Lin, Jinkun, et al.
Pubblicazione: (2025)

Accelerating Heterogeneous Tensor Parallelism via Flexible Workload Control
di: Wang, Zhigang, et al.
Pubblicazione: (2024)

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
di: Chen, Qiaoling, et al.
Pubblicazione: (2024)

Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling
di: Wang, Yujie, et al.
Pubblicazione: (2024)

Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism
di: Qing, Yuhao, et al.
Pubblicazione: (2025)

dSTAR: Straggler Tolerant and Byzantine Resilient Distributed SGD
di: Yan, Jiahe, et al.
Pubblicazione: (2024)

ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs
di: Ge, Hao, et al.
Pubblicazione: (2025)

LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
di: Liu, Xinyi, et al.
Pubblicazione: (2026)

Oases: Efficient Large-Scale Model Training on Commodity Servers via Overlapped and Automated Tensor Model Parallelism
di: Li, Shengwei, et al.
Pubblicazione: (2023)

Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training
di: Chen, Chang, et al.
Pubblicazione: (2025)

Straggler-Resilient Decentralized Learning via Adaptive Asynchronous Updates
di: Xiong, Guojun, et al.
Pubblicazione: (2023)

StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training
di: Liu, Ziming, et al.
Pubblicazione: (2024)

Fault-Tolerant Hybrid-Parallel Training at Scale with Reliable and Efficient In-memory Checkpointing
di: Wang, Yuxin, et al.
Pubblicazione: (2023)

Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training
di: Zhang, Xin, et al.
Pubblicazione: (2025)

Enhancing Memory Efficiency in Large Language Model Training Through Chronos-aware Pipeline Parallelism
di: Lin, Xinyuan, et al.
Pubblicazione: (2025)

NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism
di: Ai, Xin, et al.
Pubblicazione: (2024)

HARP: Orchestrating Automated Parallel Training on Heterogeneous GPU Clusters
di: Liang, Antian, et al.
Pubblicazione: (2025)

CFP: Efficient Optimization of Intra-Operator Parallelism Plans for Large Model Training
di: Hu, Weifang, et al.
Pubblicazione: (2025)

Communication-Efficient Serving for Video Diffusion Models with Latent Parallelism
di: Wu, Zhiyuan, et al.
Pubblicazione: (2025)

Lightweight Federated Learning with Differential Privacy and Straggler Resilience
di: Hong, Shu, et al.
Pubblicazione: (2024)

Heterogeneous Parallelism for Multimodal Large Language Model Training
di: Karnati, Yashaswi, et al.
Pubblicazione: (2026)