Imagen de Portada

Memory and Bandwidth are All You Need for Fully Sharded Data Parallel

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Wang, Jiangtao, Ebert, Jan, Filatov, Oleg, Kesselheim, Stefan
Formato:	Preprint
Publicado:	2025
Materias:	Distributed, Parallel, and Cluster Computing Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

ShardTensor: Domain Parallelism for Scientific Machine Learning
por: Adams, Corey, et al.
Publicado: (2026)

Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference
por: Yin, Ruokai, et al.
Publicado: (2025)

SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
por: Jia, Jinda, et al.
Publicado: (2024)

Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism
por: Qing, Yuhao, et al.
Publicado: (2025)

AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism
por: Ajanthan, Thalaiyasingam, et al.
Publicado: (2026)

SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile
por: Zhang, Ruisi, et al.
Publicado: (2024)

You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Models
por: Ding, Shiwei, et al.
Publicado: (2025)

Enhancing Split Learning with Sharded and Blockchain-Enabled SplitFed Approaches
por: Sokhankhosh, Amirreza, et al.
Publicado: (2025)

Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
por: Ghadia, Ravi, et al.
Publicado: (2026)

ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism
por: Aboudib, Alan, et al.
Publicado: (2026)

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
por: Tang, Zichen, et al.
Publicado: (2024)

SP-Chain: Boosting Intra-Shard and Cross-Shard Security and Performance in Blockchain Sharding
por: Li, Mingzhe, et al.
Publicado: (2024)

DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction
por: Zhang, Yanqi, et al.
Publicado: (2024)

Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator
por: Fujii, Kazuki, et al.
Publicado: (2024)

Pipeline Parallelism with Controllable Memory
por: Qi, Penghui, et al.
Publicado: (2024)

Parallelization of the K-Means Algorithm with Applications to Big Data Clustering
por: Srivastava, Ashish, et al.
Publicado: (2024)

Improving Automatic Parallel Training via Balanced Memory Workload Optimization
por: Wang, Yujie, et al.
Publicado: (2023)

Re-evaluating the Memory-balanced Pipeline Parallelism: BPipe
por: Huang, Mincong, et al.
Publicado: (2024)

Accelerating MoE Model Inference with Expert Sharding
por: Balmau, Oana, et al.
Publicado: (2025)

TASP: Topology-aware Sequence Parallelism
por: Wang, Yida, et al.
Publicado: (2025)

Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment
por: Li, Haoyang, et al.
Publicado: (2024)

DynaShard: Secure and Adaptive Blockchain Sharding Protocol with Hybrid Consensus and Dynamic Shard Management
por: Liu, Ao, et al.
Publicado: (2024)

Self-healing Nodes with Adaptive Data-Sharding
por: Thakur, Ayush, et al.
Publicado: (2024)

TRAIL: Cross-Shard Validation for Cryptocurrency Byzantine Shard Protection
por: Jacovetty, Mitch, et al.
Publicado: (2024)

From Data Center IoT Telemetry to Data Analytics Chatbots -- Virtual Knowledge Graph is All You Need
por: Khan, Junaid Ahmed, et al.
Publicado: (2025)

On Optimizing the Communication of Model Parallelism
por: Zhuang, Yonghao, et al.
Publicado: (2022)

HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
por: Zhang, Geng, et al.
Publicado: (2025)

High-Dimensional Sparse Data Low-rank Representation via Accelerated Asynchronous Parallel Stochastic Gradient Descent
por: Hu, Qicong, et al.
Publicado: (2024)

Edge-Parallel Graph Encoder Embedding
por: Lubonja, Ariel, et al.
Publicado: (2024)

Vanishing Variance Problem in Fully Decentralized Neural-Network Systems
por: Tian, Yongding, et al.
Publicado: (2024)

Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization
por: Wang, Chong, et al.
Publicado: (2026)

DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
por: Wang, Yuanqing, et al.
Publicado: (2026)

Fully Distributed Online Training of Graph Neural Networks in Networked Systems
por: Olshevskyi, Rostyslav, et al.
Publicado: (2024)

Robust Fully-Asynchronous Methods for Distributed Training over General Architecture
por: Zhu, Zehan, et al.
Publicado: (2023)

Nesterov Method for Asynchronous Pipeline Parallel Optimization
por: Ajanthan, Thalaiyasingam, et al.
Publicado: (2025)

MP-SL: Multihop Parallel Split Learning
por: Tirana, Joana, et al.
Publicado: (2024)

MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
por: Liu, Dennis, et al.
Publicado: (2025)

ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
por: Wang, Yujia, et al.
Publicado: (2025)

FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
por: Wang, Yujie, et al.
Publicado: (2024)

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
por: Chen, Qiaoling, et al.
Publicado: (2024)