Image de couverture de livre

CO2: Efficient Distributed Training with Full Communication-Computation Overlap

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Sun, Weigao, Qin, Zhen, Sun, Weixuan, Li, Shidi, Li, Dong, Shen, Xuyang, Qiao, Yu, Zhong, Yiran
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Distributed, Parallel, and Cluster Computing
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Lagom: Unleashing the Power of Communication and Computation Overlapping for Distributed LLM Training
par: Xu, Guanbin, et autres
Publié: (2026)

Cross-region Model Training with Communication-Computation Overlapping and Delay Compensation
par: Zhu, Ying, et autres
Publié: (2025)

TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives
par: Zheng, Size, et autres
Publié: (2025)

Modeling the Impact of Fiber Latency on Compute-Communication Overlap in Geo-Distributed Multi-Datacenter AI Training
par: Papavasileiou, Ioannis, et autres
Publié: (2026)

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient
par: Deng, Xiaoge, et autres
Publié: (2021)

TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference
par: Gond, Raja, et autres
Publié: (2025)

Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications
par: Lee, Seonho, et autres
Publié: (2025)

DeFT: Mitigating Data Dependencies for Flexible Communication Scheduling in Distributed Training
par: Meng, Lin, et autres
Publié: (2025)

Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping
par: Jiang, Chenyu, et autres
Publié: (2024)

Syncopate: Efficient Multi-GPU AI Kernels via Automatic Chunk-Centric Compute-Communication Overlap
par: Qiang, Xinwei, et autres
Publié: (2026)

Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering
par: Hong, Ke, et autres
Publié: (2025)

Reputation-Based Leader Election under Partial Synchrony: Towards a Protocol-Independent Abstraction with Enhanced Guarantees
par: Liu, Xuyang, et autres
Publié: (2025)

Oases: Efficient Large-Scale Model Training on Commodity Servers via Overlapped and Automated Tensor Model Parallelism
par: Li, Shengwei, et autres
Publié: (2023)

CondenseGraph: Communication-Efficient Distributed GNN Training via On-the-Fly Graph Condensation
par: Zhang, Zizhao, et autres
Publié: (2026)

AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training
par: Chen, Qiaoling, et autres
Publié: (2023)

RapidGNN: Communication Efficient Large-Scale Distributed Training of Graph Neural Networks
par: Niam, Arefin, et autres
Publié: (2025)

GreenDyGNN: Runtime-Adaptive Energy-Efficient Communication for Distributed GNN Training
par: Niam, Arefin, et autres
Publié: (2026)

Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler
par: Zheng, Size, et autres
Publié: (2025)

UNIQ: Communication-Efficient Distributed Quantum Computing via Unified Nonlinear Integer Programming
par: Zhong, Hui, et autres
Publié: (2025)

PruneX: A Hierarchical Communication-Efficient System for Distributed CNN Training with Structured Pruning
par: Olama, Alireza, et autres
Publié: (2025)

Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
par: Duan, Jiangfei, et autres
Publié: (2024)

Efficient Distributed MLLM Training with Cornstarch
par: Jang, Insu, et autres
Publié: (2025)

NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism
par: Ai, Xin, et autres
Publié: (2024)

RServe: Overlapping Encoding and Prefill for Efficient LMM Inference
par: Guo, Tianyu, et autres
Publié: (2025)

BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens
par: Sun, Ao, et autres
Publié: (2025)

Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution
par: Wang, Haiquan, et autres
Publié: (2024)

SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading
par: Chen, Qiaoling, et autres
Publié: (2025)

Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
par: Chen, Fahao, et autres
Publié: (2024)

MTGenRec: An Efficient Distributed Training System for Generative Recommendation Models in Meituan
par: Wang, Yuxiang, et autres
Publié: (2025)

ACE-Sync: An Adaptive Cloud-Edge Synchronization Framework for Communication-Efficient Large-Scale Distributed Model Training
par: Yang, Yi, et autres
Publié: (2025)

Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism
par: Qing, Yuhao, et autres
Publié: (2025)

Heta: Distributed Training of Heterogeneous Graph Neural Networks
par: Zhong, Yuchen, et autres
Publié: (2024)

Self-Evolving Distributed Memory Architecture for Scalable AI Systems
par: Li, Zixuan, et autres
Publié: (2026)

Communication-Efficient Collaborative LLM Inference over LEO Satellite Networks
par: Zhang, Songge, et autres
Publié: (2026)

DeepCompile: A Compiler-Driven Approach to Optimizing Distributed Deep Learning Training
par: Tanaka, Masahiro, et autres
Publié: (2025)

Task Scheduling in Geo-Distributed Computing: A Survey
par: Wu, Yujian, et autres
Publié: (2025)

LuWu: An End-to-End In-Network Out-of-Core Optimizer for 100B-Scale Model-in-Network Data-Parallel Training on Distributed GPUs
par: Sun, Mo, et autres
Publié: (2024)

Toward Heterogeneous, Distributed, and Energy-Efficient Computing with SYCL
par: Cosenza, Biagio, et autres
Publié: (2025)

DRust: Language-Guided Distributed Shared Memory with Fine Granularity, Full Transparency, and Ultra Efficiency
par: Ma, Haoran, et autres
Publié: (2024)

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes
par: Qin, Zhen, et autres
Publié: (2023)