Image de couverture de livre

Fault-Tolerant Hybrid-Parallel Training at Scale with Reliable and Efficient In-memory Checkpointing

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Wang, Yuxin, Kang, Xueze, Shi, Shaohuai, He, Xin, Tang, Zhenheng, Pan, Xinglin, Zheng, Yang, Wu, Xiaoyu, Zhou, Amelie Chi, He, Bingsheng, Chu, Xiaowen
Format:	Preprint
Publié:	2023
Sujets:	Distributed, Parallel, and Cluster Computing Performance
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
par: Wang, Yuxin, et autres
Publié: (2024)

DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization
par: Tang, Zhenheng, et autres
Publié: (2025)

FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression
par: Tang, Zhenheng, et autres
Publié: (2024)

Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
par: Pan, Xinglin, et autres
Publié: (2025)

Stabl: Blockchain Fault Tolerance
par: Gramoli, Vincent, et autres
Publié: (2024)

ParaLog: Consistent Host-side Logging for Parallel Checkpoints
par: Chien, Steven W. D., et autres
Publié: (2024)

GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving
par: Jayakody, Shakya, et autres
Publié: (2026)

ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training
par: Kang, Xueze, et autres
Publié: (2025)

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
par: Tang, Zichen, et autres
Publié: (2024)

ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
par: Lin, Wenxiang, et autres
Publié: (2026)

An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models
par: Chu, Xiaoyu, et autres
Publié: (2025)

DIAL: Decentralized I/O AutoTuning via Learned Client-side Local Metrics for Parallel File System
par: Rashid, Md Hasanur, et autres
Publié: (2026)

DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providers
par: Maurya, Avinash, et autres
Publié: (2026)

FAILS: A Framework for Automated Collection and Analysis of LLM Service Incidents
par: Battaglini-Fischer, Sándor, et autres
Publié: (2025)

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
par: Pan, Xinglin, et autres
Publié: (2024)

HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
par: Lin, Wenxiang, et autres
Publié: (2025)

On Orchestrating Parallel Broadcasts for Distributed Ledgers
par: Sheng, Peiyao, et autres
Publié: (2024)

Automated Programmatic Performance Analysis of Parallel Programs
par: Cankur, Onur, et autres
Publié: (2024)

Optimal Parallel Scheduling under Concave Speedup Functions
par: Li, Chengzhang, et autres
Publié: (2025)

Recorder: Comprehensive Parallel I/O Tracing and Analysis
par: Wang, Chen, et autres
Publié: (2025)

Cache Blocking of Distributed-Memory Parallel Matrix Power Kernels
par: Lacey, Dane C., et autres
Publié: (2024)

Fine-Grained Energy Prediction For Parallellized LLM Inference With PIE-P
par: Dutt, Anurag, et autres
Publié: (2025)

Automated Calibration of Parallel and Distributed Computing Simulators: A Case Study
par: McDonald, Jesse, et autres
Publié: (2024)

Efficient Fault Localization in a Cloud Stack Using End-to-End Application Service Topology
par: Mathews, Dhanya R, et autres
Publié: (2025)

RAID Organizations for Improved Reliability and Performance: A Not Entirely Unbiased Tutorial
par: Thomasian, Alexander
Publié: (2023)

Matryoshka: Optimization of Dynamic Diverse Quantum Chemistry Systems via Elastic Parallelism Transformation
par: Wang, Tuowei, et autres
Publié: (2024)

CARAT: Client-Side Adaptive RPC and Cache Co-Tuning for Parallel File Systems
par: Rashid, Md Hasanur, et autres
Publié: (2026)

HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices
par: Zhao, Xuanlei, et autres
Publié: (2024)

AcceleratedKernels.jl: Cross-Architecture Parallel Algorithms from a Unified, Transpiled Codebase
par: Nicusan, Andrei-Leonard, et autres
Publié: (2025)

Serving Chain-structured Jobs with Large Memory Footprints with Application to Large Foundation Model Serving
par: Sun, Tingyang, et autres
Publié: (2026)

HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
par: Lin, Mao, et autres
Publié: (2026)

Parallel I/O Characterization and Optimization on Large-Scale HPC Systems: A 360-Degree Survey
par: Ather, Hammad, et autres
Publié: (2024)

FastPersist: Accelerating Model Checkpointing in Deep Learning
par: Wang, Guanhua, et autres
Publié: (2024)

ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
par: Fan, Ruibo, et autres
Publié: (2026)

Efficient GPU-Centered Singular Value Decomposition Using the Divide-and-Conquer Method
par: Liu, Shifang, et autres
Publié: (2025)

Scaling Large-scale GNN Training to Thousands of Processors on CPU-based Supercomputers
par: Zhuang, Chen, et autres
Publié: (2024)

RAPID-LLM: Resilience-Aware Performance analysis of Infrastructure for Distributed LLM Training and Inference
par: Karfakis, George, et autres
Publié: (2025)

Modeling the Impact of Fiber Latency on Compute-Communication Overlap in Geo-Distributed Multi-Datacenter AI Training
par: Papavasileiou, Ioannis, et autres
Publié: (2026)

Kino-PAX: Highly Parallel Kinodynamic Sampling-based Planner
par: Perrault, Nicolas, et autres
Publié: (2024)

AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
par: Gupta, Ahan, et autres
Publié: (2026)