SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100k+ GPUs

Documenti analoghi

• TurboFFT: Co-Designed High-Performance and Fault-Tolerant Fast Fourier Transform on GPUs
  di: Wu, Shixun, et al.
  Pubblicazione: (2024)
• DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Models
  di: Maurya, Avinash, et al.
  Pubblicazione: (2024)
• ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
  di: Liu, Ziyue, et al.
  Pubblicazione: (2026)
• DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providers
  di: Maurya, Avinash, et al.
  Pubblicazione: (2026)
• Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers
  di: Maurya, Avinash, et al.
  Pubblicazione: (2024)