LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism

Documenti analoghi

• InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
  di: Chen, Qiaoling, et al.
  Pubblicazione: (2024)
• AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training
  di: Chen, Qiaoling, et al.
  Pubblicazione: (2023)
• SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading
  di: Chen, Qiaoling, et al.
  Pubblicazione: (2025)
• LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
  di: Wu, Bingyang, et al.
  Pubblicazione: (2024)
• Characterization of Large Language Model Development in the Datacenter
  di: Hu, Qinghao, et al.
  Pubblicazione: (2024)