Copertina

Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Sun, Ao, Zhao, Weilin, Han, Xu, Yang, Cheng, Zhang, Xinrong, Liu, Zhiyuan, Shi, Chuan, Sun, Maosong
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Distributed, Parallel, and Cluster Computing
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens
di: Sun, Ao, et al.
Pubblicazione: (2025)

BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences
di: Sun, Ao, et al.
Pubblicazione: (2024)

SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading
di: Chen, Qiaoling, et al.
Pubblicazione: (2025)

HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
di: Zhang, Geng, et al.
Pubblicazione: (2025)

InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training
di: Wang, Shiju, et al.
Pubblicazione: (2025)

LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
di: Gu, Diandian, et al.
Pubblicazione: (2024)

DiffusionPipe: Training Large Diffusion Models with Efficient Pipelines
di: Tian, Ye, et al.
Pubblicazione: (2024)

Bandwidth-Aware and Cost-Efficient Pipeline Parallel Scheduling in Geo-Distributed LLM Training
di: Zhang, Han, et al.
Pubblicazione: (2026)

DawnPiper: A Memory-scablable Pipeline Parallel Training Framework
di: Peng, Xuan, et al.
Pubblicazione: (2025)

StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training
di: Liu, Ziming, et al.
Pubblicazione: (2024)

Enhancing Memory Efficiency in Large Language Model Training Through Chronos-aware Pipeline Parallelism
di: Lin, Xinyuan, et al.
Pubblicazione: (2025)

A Flexible Programmable Pipeline Parallelism Framework for Efficient DNN Training
di: Jiang, Lijuan, et al.
Pubblicazione: (2025)

JanusPipe: Efficient Pipeline Parallel Training for Machine Learning Interatomic Potentials
di: Wang, Hongyu, et al.
Pubblicazione: (2026)

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
di: Chen, Qiaoling, et al.
Pubblicazione: (2024)

HexiSeq: Accommodating Long Context Training of LLMs over Heterogeneous Hardware
di: Liang, Yan, et al.
Pubblicazione: (2026)

Memory Efficient and Staleness Free Pipeline Parallel DNN Training Framework with Improved Convergence Speed
di: Dutta, Ankita, et al.
Pubblicazione: (2025)

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
di: Wu, Bingyang, et al.
Pubblicazione: (2024)

Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation
di: Wu, Tianyuan, et al.
Pubblicazione: (2025)

TiMePReSt: Time and Memory Efficient Pipeline Parallel DNN Training with Removed Staleness
di: Dutta, Ankita, et al.
Pubblicazione: (2024)

CFP: Efficient Optimization of Intra-Operator Parallelism Plans for Large Model Training
di: Hu, Weifang, et al.
Pubblicazione: (2025)

Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models
di: Wang, Wei, et al.
Pubblicazione: (2024)

Balancing Pipeline Parallelism with Vocabulary Parallelism
di: Yeung, Man Tsung, et al.
Pubblicazione: (2024)

SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference
di: He, Yongchao, et al.
Pubblicazione: (2025)

Synergistic Tensor and Pipeline Parallelism
di: Qi, Mengshi, et al.
Pubblicazione: (2025)

MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism
di: Zhang, Zheng, et al.
Pubblicazione: (2025)

Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism
di: Qing, Yuhao, et al.
Pubblicazione: (2025)

Heimdall++: Optimizing GPU Utilization and Pipeline Parallelism for Efficient Single-Pulse Detection
di: Xia, Bingzheng, et al.
Pubblicazione: (2025)

Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training
di: Chen, Chang, et al.
Pubblicazione: (2025)

TrioSeq: A Novel Approach to Accelerate Triplet Sequence Alignment on GPUs
di: Graça, Miguel, et al.
Pubblicazione: (2026)

FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
di: Wang, Yujie, et al.
Pubblicazione: (2024)

MSPipe: Efficient Temporal GNN Training via Staleness-Aware Pipeline
di: Sheng, Guangming, et al.
Pubblicazione: (2024)

Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
di: Duan, Jiangfei, et al.
Pubblicazione: (2024)

AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
di: Chen, Ling, et al.
Pubblicazione: (2026)

COPUS: Co-adaptive Parallelism and Batch Size Selection in Large Language Model Training
di: Sakip, Akhmed, et al.
Pubblicazione: (2026)

Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
di: Shyam, Vasu, et al.
Pubblicazione: (2026)

PPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via Pool-Based Pipeline Parallelism
di: Kong, Z. Jonny, et al.
Pubblicazione: (2025)

Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism
di: Li, Cong, et al.
Pubblicazione: (2025)

AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models
di: Guo, Jihu, et al.
Pubblicazione: (2025)

HARP: Orchestrating Automated Parallel Training on Heterogeneous GPU Clusters
di: Liang, Antian, et al.
Pubblicazione: (2025)

Oases: Efficient Large-Scale Model Training on Commodity Servers via Overlapped and Automated Tensor Model Parallelism
di: Li, Shengwei, et al.
Pubblicazione: (2023)