HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism

Documenti analoghi

• DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
  di: Zhao, Xuanlei, et al.
  Pubblicazione: (2024)
• StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training
  di: Liu, Ziming, et al.
  Pubblicazione: (2024)
• HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices
  di: Zhao, Xuanlei, et al.
  Pubblicazione: (2024)
• AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference
  di: Zhao, Xuanlei, et al.
  Pubblicazione: (2024)
• ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
  di: Liu, Zedong, et al.
  Pubblicazione: (2025)