Cover Image

SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Ruisi, Liu, Tianyu, Feng, Will, Gu, Andrew, Purandare, Sanket, Liang, Wanchao, Massa, Francisco
Format:	Preprint
Published:	2024
Subjects:	Distributed, Parallel, and Cluster Computing Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism
by: Qing, Yuhao, et al.
Published: (2025)

Memory and Bandwidth are All You Need for Fully Sharded Data Parallel
by: Wang, Jiangtao, et al.
Published: (2025)

veScale-FSDP: Flexible and High-Performance FSDP at Scale
by: Wang, Zezhou, et al.
Published: (2026)

DynaShard: Secure and Adaptive Blockchain Sharding Protocol with Hybrid Consensus and Dynamic Shard Management
by: Liu, Ao, et al.
Published: (2024)

SP-Chain: Boosting Intra-Shard and Cross-Shard Security and Performance in Blockchain Sharding
by: Li, Mingzhe, et al.
Published: (2024)

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
by: Chen, Qiaoling, et al.
Published: (2024)

Self-healing Nodes with Adaptive Data-Sharding
by: Thakur, Ayush, et al.
Published: (2024)

TRAIL: Cross-Shard Validation for Cryptocurrency Byzantine Shard Protection
by: Jacovetty, Mitch, et al.
Published: (2024)

StableShard: Stable and Scalable Blockchain Sharding with High Concurrency via Collaborative Committees
by: Li, Mingzhe, et al.
Published: (2024)

ShardTensor: Domain Parallelism for Scientific Machine Learning
by: Adams, Corey, et al.
Published: (2026)

TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
by: Liang, Wanchao, et al.
Published: (2024)

Performance Characterization of Distributed Deep Learning Strategies: A Quantitative Evaluation of DDP, FSDP, and Parameter Server Architectures on GPU Clusters
by: Ovi, Md Sultanul Islam
Published: (2025)

SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
by: Jia, Jinda, et al.
Published: (2024)

Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference
by: Yin, Ruokai, et al.
Published: (2025)

Fast Transaction Scheduling in Blockchain Sharding
by: Adhikari, Ramesh, et al.
Published: (2024)

On the Efficiency of Dynamic Transaction Scheduling in Blockchain Sharding
by: Adhikari, Ramesh, et al.
Published: (2025)

Sharding Distributed Databases: A Critical Review
by: Solat, Siamak
Published: (2024)

torch-sla: Differentiable Sparse Linear Algebra with Adjoint Solvers and Sparse Tensor Parallelism for PyTorch
by: Chi, Mingyuan, et al.
Published: (2026)

A Hierarchical Sharded Blockchain Balancing Performance and Availability
by: Jo, Yongrae, et al.
Published: (2025)

Dynamically Sharded Ledgers on a Distributed Hash Table
by: Fink, Christoffer, et al.
Published: (2024)

Stable Blockchain Sharding under Adversarial Transaction Generation
by: Adhikari, Ramesh, et al.
Published: (2024)

Investigating Sharding Advancements, Methodologies, and Adoption Potential in Hedera
by: Wang, Ziwei, et al.
Published: (2025)

Near-Optimal Stability for Distributed Transaction Processing in Blockchain Sharding
by: Adhikari, Ramesh, et al.
Published: (2025)

SmartShards: Churn-Tolerant Continuously Available Distributed Ledger
by: Oglio, Joseph, et al.
Published: (2025)

Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
by: Bhatia, Nidhi, et al.
Published: (2025)

HiCoCS: High Concurrency Cross-Sharding on Permissioned Blockchains
by: Yang, Lingxiao, et al.
Published: (2025)

EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
by: Zhang, Mingjin, et al.
Published: (2024)

Automated Market Makers for Cross-chain DeFi and Sharded Blockchains
by: Aanes, Jon Michael, et al.
Published: (2023)

AI-driven Predictive Shard Allocation for Scalable Next Generation Blockchains
by: Haider, M. Zeeshan, et al.
Published: (2025)

The Merit of Simple Policies: Buying Performance With Parallelism and System Architecture
by: Yildiz, Mert, et al.
Published: (2025)

AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism
by: Ajanthan, Thalaiyasingam, et al.
Published: (2026)

Advancing Blockchain Scalability: A Linear Optimization Framework for Diversified Node Allocation in Shards
by: Assmann, Björn, et al.
Published: (2024)

Fully integrating the Flang Fortran compiler with standard MLIR
by: Brown, Nick
Published: (2024)

SpiralShard: Highly Concurrent and Secure Blockchain Sharding via Linked Cross-shard Endorsement
by: Lin, You, et al.
Published: (2024)

Enhancing IoT Malware Detection through Adaptive Model Parallelism and Resource Optimization
by: Kasarapu, Sreenitha, et al.
Published: (2024)

Highly Dynamic and Fully Distributed Data Structures
by: Augustine, John, et al.
Published: (2024)

SAMM: Sharded Automated Market Maker
by: Chen, Hongyin, et al.
Published: (2024)

NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decoding
by: Chen, Jiefei, et al.
Published: (2026)

gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling
by: Guo, Tianyu, et al.
Published: (2025)

Parallel Writing of Nested Data in Columnar Formats
by: Hahnfeld, Jonas, et al.
Published: (2024)