Buchumschlag

DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models

Fuente: arXiv

Gespeichert in:

Bibliographische Detailangaben
Hauptverfasser:	Zhang, Zili, Zhong, Yinmin, Jiang, Yimin, Hu, Hanpeng, Sun, Jianjian, Ge, Zheng, Zhu, Yibo, Jiang, Daxin, Jin, Xin
Format:	Preprint
Veröffentlicht:	2024
Schlagworte:	Distributed, Parallel, and Cluster Computing
Online-Zugang:	Acceder al recurso
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Ähnliche Einträge

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
von: Zhong, Yinmin, et al.
Veröffentlicht: (2024)

StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
von: Zhong, Yinmin, et al.
Veröffentlicht: (2025)

DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipeline
von: Xue, Zhenliang, et al.
Veröffentlicht: (2025)

Optimizing RLHF Training for Large Language Models with Stage Fusion
von: Zhong, Yinmin, et al.
Veröffentlicht: (2024)

Efficient Long-context Language Model Training by Core Attention Disaggregation
von: Zhuang, Yonghao, et al.
Veröffentlicht: (2025)

Accelerating Distributed MoE Training and Inference with Lina
von: Li, Jiamin, et al.
Veröffentlicht: (2022)

Addressing Variable Heterogeneity in Distributed Multimodal Training with Entrain
von: Jang, Insu, et al.
Veröffentlicht: (2026)

DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training
von: Tan, Xin, et al.
Veröffentlicht: (2025)

TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
von: Wu, Bingyang, et al.
Veröffentlicht: (2025)

OrchestrRL: Dynamic Compute and Network Orchestration for Disaggregated RL
von: Tan, Xin, et al.
Veröffentlicht: (2026)

Fast Distributed Inference Serving for Large Language Models
von: Wu, Bingyang, et al.
Veröffentlicht: (2023)

HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware
von: Yan, Ran, et al.
Veröffentlicht: (2024)

Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation
von: Chen, Shaoyuan, et al.
Veröffentlicht: (2024)

HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment
von: Jiang, Youhe, et al.
Veröffentlicht: (2025)

Heterogeneous Parallelism for Multimodal Large Language Model Training
von: Karnati, Yashaswi, et al.
Veröffentlicht: (2026)

HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving
von: Dong, Xianzhe, et al.
Veröffentlicht: (2025)

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
von: Wu, Bingyang, et al.
Veröffentlicht: (2024)

DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training
von: Wang, Zhixin, et al.
Veröffentlicht: (2025)

Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
von: Duan, Jiangfei, et al.
Veröffentlicht: (2024)

vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models
von: Yin, Peiqi, et al.
Veröffentlicht: (2026)

RollMux: Phase-Level Multiplexing for Disaggregated RL Post-Training
von: Wu, Tianyuan, et al.
Veröffentlicht: (2025)

AcOrch: Accelerating Sampling-based GNN Training under CPU-NPU Heterogeneous Environments
von: Chen, Kefu, et al.
Veröffentlicht: (2026)

Heta: Distributed Training of Heterogeneous Graph Neural Networks
von: Zhong, Yuchen, et al.
Veröffentlicht: (2024)

Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs
von: Chen, Xing, et al.
Veröffentlicht: (2025)

ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale
von: Won, William, et al.
Veröffentlicht: (2023)

Training DNN Models over Heterogeneous Clusters with Optimal Performance
von: Nie, Chengyi, et al.
Veröffentlicht: (2024)

Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models
von: Guo, Runsheng Benson, et al.
Veröffentlicht: (2024)

MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
von: Jin, Chao, et al.
Veröffentlicht: (2025)

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
von: Jiang, Ziheng, et al.
Veröffentlicht: (2024)

HexiSeq: Accommodating Long Context Training of LLMs over Heterogeneous Hardware
von: Liang, Yan, et al.
Veröffentlicht: (2026)

Proceedings of 3rd Workshop on Heterogeneous Composable and Disaggregated Systems
von: Pinto, Christian, et al.
Veröffentlicht: (2024)

Oases: Efficient Large-Scale Model Training on Commodity Servers via Overlapped and Automated Tensor Model Parallelism
von: Li, Shengwei, et al.
Veröffentlicht: (2023)

InfiniLoRA: Disaggregated Multi-LoRA Serving for Large Language Models
von: Chen, Hongyu, et al.
Veröffentlicht: (2026)

Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization
von: Li, Haoyang, et al.
Veröffentlicht: (2024)

Mosaic: Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing
von: Wang, Yanbo, et al.
Veröffentlicht: (2026)

Galvatron: Automatic Distributed Training for Large Transformer Models
von: Gumaan, Esmail
Veröffentlicht: (2025)

MTGenRec: An Efficient Distributed Training System for Generative Recommendation Models in Meituan
von: Wang, Yuxiang, et al.
Veröffentlicht: (2025)

GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions
von: Shi, Tianyao, et al.
Veröffentlicht: (2024)

Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters
von: Zhang, WenZheng, et al.
Veröffentlicht: (2024)

CFP: Efficient Optimization of Intra-Operator Parallelism Plans for Large Model Training
von: Hu, Weifang, et al.
Veröffentlicht: (2025)