Cover Image

Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhu, Zhanda, Giannoula, Christina, Andoorveedu, Muralidhar, Su, Qidong, Mangalam, Karttikeya, Zheng, Bojian, Pekhimenko, Gennady
Format:	Preprint
Published:	2025
Subjects:	Distributed, Parallel, and Cluster Computing Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Seesaw: High-throughput LLM Inference via Model Re-sharding
by: Su, Qidong, et al.
Published: (2025)

Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agents
by: Song, Kevin, et al.
Published: (2025)

LoRAFusion: Efficient LoRA Fine-Tuning for LLMs
by: Zhu, Zhanda, et al.
Published: (2025)

DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures
by: Yang, Peiming, et al.
Published: (2025)

Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads
by: Zhao, Wei, et al.
Published: (2024)

PyGim: An Efficient Graph Neural Network Library for Real Processing-In-Memory Architectures
by: Giannoula, Christina, et al.
Published: (2024)

DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling
by: Gao, Yubo, et al.
Published: (2025)

Minuet: Accelerating 3D Sparse Convolutions on GPUs
by: Yang, Jiacheng, et al.
Published: (2023)

CoCoDiff: Optimizing Collective Communications for Distributed Diffusion Transformer Inference Under Ulysses Sequence Parallelism
by: Ma, Bin, et al.
Published: (2026)

CFP: Efficient Optimization of Intra-Operator Parallelism Plans for Large Model Training
by: Hu, Weifang, et al.
Published: (2025)

SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs
by: Yang, Jiacheng, et al.
Published: (2026)

HybridTier: an Adaptive and Lightweight CXL-Memory Tiering System
by: Song, Kevin, et al.
Published: (2023)

SmartPQ: An Adaptive Concurrent Priority Queue for NUMA Architectures
by: Giannoula, Christina, et al.
Published: (2024)

Efficient Parallelization Layouts for Large-Scale Distributed Model Training
by: Hagemann, Johannes, et al.
Published: (2023)

Proceedings of 3rd Workshop on Heterogeneous Composable and Disaggregated Systems
by: Pinto, Christian, et al.
Published: (2024)

Memory Efficient and Staleness Free Pipeline Parallel DNN Training Framework with Improved Convergence Speed
by: Dutta, Ankita, et al.
Published: (2025)

Enhancing Memory Efficiency in Large Language Model Training Through Chronos-aware Pipeline Parallelism
by: Lin, Xinyuan, et al.
Published: (2025)

COPUS: Co-adaptive Parallelism and Batch Size Selection in Large Language Model Training
by: Sakip, Akhmed, et al.
Published: (2026)

Bandwidth-Aware and Cost-Efficient Pipeline Parallel Scheduling in Geo-Distributed LLM Training
by: Zhang, Han, et al.
Published: (2026)

TiMePReSt: Time and Memory Efficient Pipeline Parallel DNN Training with Removed Staleness
by: Dutta, Ankita, et al.
Published: (2024)

Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models
by: Wang, Wei, et al.
Published: (2024)

DawnPiper: A Memory-scablable Pipeline Parallel Training Framework
by: Peng, Xuan, et al.
Published: (2025)

GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching
by: Guo, Cong, et al.
Published: (2024)

LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
by: Gu, Diandian, et al.
Published: (2024)

MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism
by: Zhang, Zheng, et al.
Published: (2025)

SiDP: Memory-Efficient Data Parallelism for Offline LLM Inference
by: Zhao, Alan, et al.
Published: (2026)

CaPGNN: Optimizing Parallel Graph Neural Network Training with Joint Caching and Resource-Aware Graph Partitioning
by: Song, Xianfeng, et al.
Published: (2025)

Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
by: Duan, Jiangfei, et al.
Published: (2024)

Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design
by: Xue, Chunyu, et al.
Published: (2024)

Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
by: Shyam, Vasu, et al.
Published: (2026)

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
by: Chen, Qiaoling, et al.
Published: (2024)

Oases: Efficient Large-Scale Model Training on Commodity Servers via Overlapped and Automated Tensor Model Parallelism
by: Li, Shengwei, et al.
Published: (2023)

Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training
by: Sun, Ao, et al.
Published: (2024)

Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networks
by: Waleffe, Roger, et al.
Published: (2025)

Distributed-Memory Parallel Algorithms for Sparse Matrix and Sparse Tall-and-Skinny Matrix Multiplication
by: Ranawaka, Isuru, et al.
Published: (2024)

Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters
by: Zhang, WenZheng, et al.
Published: (2024)

A Flexible Programmable Pipeline Parallelism Framework for Efficient DNN Training
by: Jiang, Lijuan, et al.
Published: (2025)

Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization
by: Li, Haoyang, et al.
Published: (2024)

On the Performance and Memory Footprint of Distributed Training: An Empirical Study on Transformers
by: Lu, Zhengxian, et al.
Published: (2024)

Efficient Data-Parallel Continual Learning with Asynchronous Distributed Rehearsal Buffers
by: Bouvier, Thomas, et al.
Published: (2024)