ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs

Documenti analoghi

• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
  di: Jiang, Ziheng, et al.
  Pubblicazione: (2024)
• HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware
  di: Yan, Ran, et al.
  Pubblicazione: (2024)
• Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization
  di: Li, Haoyang, et al.
  Pubblicazione: (2024)
• Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs
  di: He, Guoliang, et al.
  Pubblicazione: (2025)
• LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing
  di: Nie, Xiaonan, et al.
  Pubblicazione: (2024)