Imagen de Portada

StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Liu, Ziming, Wang, Shaoyu, Cheng, Shenggan, Zhao, Zhongkai, Wang, Kai, Zhao, Xuanlei, Demmel, James, You, Yang
Formato:	Preprint
Publicado:	2024
Materias:	Distributed, Parallel, and Cluster Computing
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
por: Zhang, Geng, et al.
Publicado: (2025)

DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
por: Zhao, Xuanlei, et al.
Publicado: (2024)

HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices
por: Zhao, Xuanlei, et al.
Publicado: (2024)

AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference
por: Zhao, Xuanlei, et al.
Publicado: (2024)

LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
por: Gu, Diandian, et al.
Publicado: (2024)

TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication
por: Wang, Zongwu, et al.
Publicado: (2024)

Real-Time Video Generation with Pyramid Attention Broadcast
por: Zhao, Xuanlei, et al.
Publicado: (2024)

ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
por: Liu, Zedong, et al.
Publicado: (2025)

Large Scale Multi-GPU Based Parallel Traffic Simulation for Accelerated Traffic Assignment and Propagation
por: Jiang, Xuan, et al.
Publicado: (2024)

HARP: Orchestrating Automated Parallel Training on Heterogeneous GPU Clusters
por: Liang, Antian, et al.
Publicado: (2025)

Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP
por: Zhao, Yilong, et al.
Publicado: (2026)

Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training
por: Sun, Ao, et al.
Publicado: (2024)

Amoeba: Runtime Tensor Parallel Transformation for LLM Inference Services
por: Chen, Haoyu, et al.
Publicado: (2025)

DiT-HC: Enabling Efficient Training of Visual Generation Model DiT on HPC-oriented CPU Cluster
por: Zhang, Jinxiao, et al.
Publicado: (2026)

Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
por: Shyam, Vasu, et al.
Publicado: (2026)

BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens
por: Sun, Ao, et al.
Publicado: (2025)

SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading
por: Chen, Qiaoling, et al.
Publicado: (2025)

DawnPiper: A Memory-scablable Pipeline Parallel Training Framework
por: Peng, Xuan, et al.
Publicado: (2025)

AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
por: Gupta, Ahan, et al.
Publicado: (2026)

NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decoding
por: Chen, Jiefei, et al.
Publicado: (2026)

FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training
por: Qi, Shuyao, et al.
Publicado: (2026)

Chameleon: Taming Dynamic Operator Sequences for Memory-Intensive LLM Training
por: Wang, Zibo, et al.
Publicado: (2025)

Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization
por: Li, Haoyang, et al.
Publicado: (2024)

CoCoDiff: Optimizing Collective Communications for Distributed Diffusion Transformer Inference Under Ulysses Sequence Parallelism
por: Ma, Bin, et al.
Publicado: (2026)

Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism
por: Qing, Yuhao, et al.
Publicado: (2025)

DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism
por: Jiang, Chenyu, et al.
Publicado: (2025)

Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation
por: Wu, Tianyuan, et al.
Publicado: (2025)

OMP-Engineer: Bridging Syntax Analysis and In-Context Learning for Efficient Automated OpenMP Parallelization
por: Wang, Weidong, et al.
Publicado: (2024)

TASP: Topology-aware Sequence Parallelism
por: Wang, Yida, et al.
Publicado: (2025)

JanusPipe: Efficient Pipeline Parallel Training for Machine Learning Interatomic Potentials
por: Wang, Hongyu, et al.
Publicado: (2026)

FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
por: Wang, Yujie, et al.
Publicado: (2024)

ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism
por: Ma, Tenghui, et al.
Publicado: (2026)

Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving
por: Liu, Ziming, et al.
Publicado: (2025)

ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training
por: Kang, Xueze, et al.
Publicado: (2025)

Parallel DNA Sequence Alignment on High-Performance Systems with CUDA and MPI
por: Zwaka, Linus
Publicado: (2024)

InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training
por: Wang, Shiju, et al.
Publicado: (2025)

Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training
por: Chen, Chang, et al.
Publicado: (2025)

Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism
por: Li, Cong, et al.
Publicado: (2025)

SiDP: Memory-Efficient Data Parallelism for Offline LLM Inference
por: Zhao, Alan, et al.
Publicado: (2026)

NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism
por: Ai, Xin, et al.
Publicado: (2024)