StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training

Documenti analoghi

• HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
  di: Zhang, Geng, et al.
  Pubblicazione: (2025)
• DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
  di: Zhao, Xuanlei, et al.
  Pubblicazione: (2024)
• HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices
  di: Zhao, Xuanlei, et al.
  Pubblicazione: (2024)
• AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference
  di: Zhao, Xuanlei, et al.
  Pubblicazione: (2024)
• LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
  di: Gu, Diandian, et al.
  Pubblicazione: (2024)