Image de couverture de livre

Re-evaluating the Memory-balanced Pipeline Parallelism: BPipe

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Huang, Mincong, Wang, Chao, Ma, Chi, Zhang, Yineng, Zhang, Peng, Yu, Lei
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Computation and Language Distributed, Parallel, and Cluster Computing
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Pipeline Parallelism with Controllable Memory
par: Qi, Penghui, et autres
Publié: (2024)

Scaling Deep Learning Training with MPMD Pipeline Parallelism
par: Xhebraj, Anxhelo, et autres
Publié: (2024)

PiPar: Pipeline Parallelism for Collaborative Machine Learning
par: Zhang, Zihan, et autres
Publié: (2022)

HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
par: Zhang, Geng, et autres
Publié: (2025)

PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation
par: Butler, Branden, et autres
Publié: (2024)

PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
par: Wan, Xinyi, et autres
Publié: (2025)

Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
par: Cai, Weilin, et autres
Publié: (2024)

Nesterov Method for Asynchronous Pipeline Parallel Optimization
par: Ajanthan, Thalaiyasingam, et autres
Publié: (2025)

AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
par: Chen, Ling, et autres
Publié: (2026)

A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability
par: Liu, Ruitao, et autres
Publié: (2026)

AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism
par: Ajanthan, Thalaiyasingam, et autres
Publié: (2026)

Synergistic Tensor and Pipeline Parallelism
par: Qi, Mengshi, et autres
Publié: (2025)

DawnPiper: A Memory-scablable Pipeline Parallel Training Framework
par: Peng, Xuan, et autres
Publié: (2025)

PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving
par: Bai, Xu, et autres
Publié: (2026)

Zero Bubble Pipeline Parallelism
par: Qi, Penghui, et autres
Publié: (2023)

MSPipe: Efficient Temporal GNN Training via Staleness-Aware Pipeline
par: Sheng, Guangming, et autres
Publié: (2024)

ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation
par: Gandhi, Swapnil, et autres
Publié: (2024)

DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction
par: Zhang, Yanqi, et autres
Publié: (2024)

Memory and Bandwidth are All You Need for Fully Sharded Data Parallel
par: Wang, Jiangtao, et autres
Publié: (2025)

JORA: JAX Tensor-Parallel LoRA Library for Retrieval Augmented Fine-Tuning
par: Tahir, Anique, et autres
Publié: (2024)

A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
par: Barley, Daniel, et autres
Publié: (2026)

Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding
par: Jin, Tian, et autres
Publié: (2025)

semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
par: Hong, Ke, et autres
Publié: (2025)

RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
par: Jin, Chao, et autres
Publié: (2024)

Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering
par: Hong, Ke, et autres
Publié: (2025)

DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
par: Wang, Yuanqing, et autres
Publié: (2026)

Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
par: Ghadia, Ravi, et autres
Publié: (2026)

Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design
par: Zhang, Mohan, et autres
Publié: (2025)

MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
par: Zhu, Ruidong, et autres
Publié: (2025)

P/D-Device: Disaggregated Large Language Model between Cloud and Devices
par: Jin, Yibo, et autres
Publié: (2025)

TASP: Topology-aware Sequence Parallelism
par: Wang, Yida, et autres
Publié: (2025)

Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference
par: Luo, Shuqing, et autres
Publié: (2025)

Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator
par: Fujii, Kazuki, et autres
Publié: (2024)

Hermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices
par: Han, Xueyuan, et autres
Publié: (2024)

PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training
par: Arfeen, Daiyaan, et autres
Publié: (2024)

ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems
par: Chen, Qiaoling, et autres
Publié: (2025)

InferCept: Efficient Intercept Support for Augmented Large Language Model Inference
par: Abhyankar, Reyna, et autres
Publié: (2024)

Optimizing RLHF Training for Large Language Models with Stage Fusion
par: Zhong, Yinmin, et autres
Publié: (2024)

X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
par: Yuan, Yueming, et autres
Publié: (2025)

Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models
par: Pan, Xuchen, et autres
Publié: (2025)