Cover Image

OptPipe: Memory- and Scheduling-Optimized Pipeline Parallelism for LLM Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Hongpei, Zhang, Han, Liu, Huikang, Ge, Dongdong, Ye, Yinyu
Format:	Preprint
Published:	2025
Subjects:	Distributed, Parallel, and Cluster Computing Artificial Intelligence Optimization and Control
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

D-PDLP: Scaling PDLP to Distributed Multi-GPU Systems
by: Li, Hongpei, et al.
Published: (2026)

Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
by: Ao, Ruicheng, et al.
Published: (2025)

InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training
by: Wang, Shiju, et al.
Published: (2025)

Constraint Programming Models For Serial Batch Scheduling With Minimum Batch Size
by: Huertas, Jorge A., et al.
Published: (2025)

PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
by: Wan, Xinyi, et al.
Published: (2025)

Resource Allocation for Stable LLM Training in Mobile Edge Computing
by: Liu, Chang, et al.
Published: (2024)

Distributed Difference of Convex Optimization
by: Khatana, Vivek, et al.
Published: (2024)

Communication Efficient Distributed Training with Distributed Lion
by: Liu, Bo, et al.
Published: (2024)

GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism
by: Jeon, Byungsoo, et al.
Published: (2024)

BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Training
by: Wu, Houming, et al.
Published: (2024)

From Sequential to Parallel: Reformulating Dynamic Programming as GPU Kernels for Large-Scale Stochastic Combinatorial Optimization
by: Zhao, Jingyi, et al.
Published: (2026)

MAST: Model-Agnostic Sparsified Training
by: Demidovich, Yury, et al.
Published: (2023)

A Communication and Computation Efficient Fully First-order Method for Decentralized Bilevel Optimization
by: Wen, Min, et al.
Published: (2024)

FADAS: Towards Federated Adaptive Asynchronous Optimization
by: Wang, Yujia, et al.
Published: (2024)

Correlated Quantization for Faster Nonconvex Distributed Optimization
by: Panferov, Andrei, et al.
Published: (2024)

Several Performance Bounds on Decentralized Online Optimization are Highly Conservative and Potentially Misleading
by: Meunier, Erwan, et al.
Published: (2025)

TawPipe: Topology-Aware Weight Pipeline Parallelism for Accelerating Long-Context Large Models Training
by: Wu, Houming, et al.
Published: (2025)

Large-Scale LLM Inference with Heterogeneous Workloads: Prefill-Decode Contention and Asymptotically Optimal Control
by: Lin, Ruihan, et al.
Published: (2026)

Bandwidth-Aware and Cost-Efficient Pipeline Parallel Scheduling in Geo-Distributed LLM Training
by: Zhang, Han, et al.
Published: (2026)

Activations and Gradients Compression for Model-Parallel Training
by: Rudakov, Mikhail, et al.
Published: (2024)

N2N: A Parallel Framework for Large-Scale MILP under Distributed Memory
by: Wang, Longfei, et al.
Published: (2025)

Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization
by: Zhu, Zhanda, et al.
Published: (2025)

Decentralized Distributed Optimization for Saddle Point Problems
by: Rogozin, Alexander, et al.
Published: (2021)

Decentralized Nonsmooth Nonconvex Optimization with Client Sampling
by: Chen, Xinyan, et al.
Published: (2026)

CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training
by: Chen, Tiancheng, et al.
Published: (2025)

FreeRide: Harvesting Bubbles in Pipeline Parallelism
by: Zhang, Jiashu, et al.
Published: (2024)

An Optimistic Gradient Tracking Method for Distributed Minimax Optimization
by: Huang, Yan, et al.
Published: (2025)

Tailoring Gradient Methods for Differentially-Private Distributed Optimization
by: Wang, Yongqiang, et al.
Published: (2022)

Problem-Parameter-Free Decentralized Nonconvex Stochastic Optimization
by: Li, Jiaxiang, et al.
Published: (2024)

JanusPipe: Efficient Pipeline Parallel Training for Machine Learning Interatomic Potentials
by: Wang, Hongyu, et al.
Published: (2026)

Distributed Constraint-Coupled Optimization: Harnessing ADMM-consensus for robustness
by: Messilem, Mohamed Abdelmouamin, et al.
Published: (2025)

Distributed Constrained Combinatorial Optimization leveraging Hypergraph Neural Networks
by: Heydaribeni, Nasimeh, et al.
Published: (2023)

TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelism
by: Cho, Seonghye, et al.
Published: (2026)

One-Point Feedback for Composite Optimization with Applications to Distributed and Federated Learning
by: Beznosikov, Aleksandr, et al.
Published: (2021)

Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods
by: Begunov, Grigory, et al.
Published: (2026)

Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences
by: Malinovsky, Grigory, et al.
Published: (2023)

PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding
by: McDanel, Bradley, et al.
Published: (2025)

Decentralized Gradient-Free Methods for Stochastic Non-Smooth Non-Convex Optimization
by: Lin, Zhenwei, et al.
Published: (2023)

Decentralized Nonconvex Optimization under Heavy-Tailed Noise: Normalization and Optimal Convergence
by: Yu, Shuhua, et al.
Published: (2025)

Smoothed Gradient Clipping and Error Feedback for Decentralized Optimization under Symmetric Heavy-Tailed Noise
by: Yu, Shuhua, et al.
Published: (2023)