Copertina

Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Qing, Yuhao, Zhu, Guichao, Li, Fanxin, Lei, Lintian, Sun, Zekai, Guan, Xiuxian, Zhao, Shixiong, Chen, Xusheng, Huang, Dong, Wang, Sen, Cui, Heming
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Distributed, Parallel, and Cluster Computing
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Hybrid-Parallel: Achieving High Performance and Energy Efficient Distributed Inference on Robots
di: Sun, Zekai, et al.
Pubblicazione: (2024)

SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
di: Zhou, Qihui, et al.
Pubblicazione: (2025)

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
di: Chen, Qiaoling, et al.
Pubblicazione: (2024)

SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile
di: Zhang, Ruisi, et al.
Pubblicazione: (2024)

Memory and Bandwidth are All You Need for Fully Sharded Data Parallel
di: Wang, Jiangtao, et al.
Pubblicazione: (2025)

EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
di: Zhang, Mingjin, et al.
Pubblicazione: (2024)

Distributed-Memory Parallel Algorithms for Sparse Matrix and Sparse Tall-and-Skinny Matrix Multiplication
di: Ranawaka, Isuru, et al.
Pubblicazione: (2024)

SP-Chain: Boosting Intra-Shard and Cross-Shard Security and Performance in Blockchain Sharding
di: Li, Mingzhe, et al.
Pubblicazione: (2024)

DynaShard: Secure and Adaptive Blockchain Sharding Protocol with Hybrid Consensus and Dynamic Shard Management
di: Liu, Ao, et al.
Pubblicazione: (2024)

Accelerating Sparse DNNs Based on Tiled GEMM
di: Guo, Cong, et al.
Pubblicazione: (2024)

TRAIL: Cross-Shard Validation for Cryptocurrency Byzantine Shard Protection
di: Jacovetty, Mitch, et al.
Pubblicazione: (2024)

SparDL: Distributed Deep Learning Training with Efficient Sparse Communication
di: Zhao, Minjun, et al.
Pubblicazione: (2023)

Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
di: Chen, Fahao, et al.
Pubblicazione: (2024)

AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
di: Liu, Di, et al.
Pubblicazione: (2026)

StableShard: Stable and Scalable Blockchain Sharding with High Concurrency via Collaborative Committees
di: Li, Mingzhe, et al.
Pubblicazione: (2024)

Sparse Checkpointing for Fast and Reliable MoE Training
di: Gandhi, Swapnil, et al.
Pubblicazione: (2024)

ShardTensor: Domain Parallelism for Scientific Machine Learning
di: Adams, Corey, et al.
Pubblicazione: (2026)

Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference
di: Yin, Ruokai, et al.
Pubblicazione: (2025)

SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
di: Jia, Jinda, et al.
Pubblicazione: (2024)

Fast Transaction Scheduling in Blockchain Sharding
di: Adhikari, Ramesh, et al.
Pubblicazione: (2024)

Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training
di: Zhang, Xin, et al.
Pubblicazione: (2025)

Self-healing Nodes with Adaptive Data-Sharding
di: Thakur, Ayush, et al.
Pubblicazione: (2024)

On the Efficiency of Dynamic Transaction Scheduling in Blockchain Sharding
di: Adhikari, Ramesh, et al.
Pubblicazione: (2025)

Sharding Distributed Databases: A Critical Review
di: Solat, Siamak
Pubblicazione: (2024)

AsyncSparse: Accelerating Sparse Matrix-Matrix Multiplication on Asynchronous GPU Architectures
di: Liu, Jie, et al.
Pubblicazione: (2026)

PARS3: Parallel Sparse Skew-Symmetric Matrix-Vector Multiplication with Reverse Cuthill-McKee Reordering
di: Yildirim, Selin, et al.
Pubblicazione: (2024)

Selection of Supervised Learning-based Sparse Matrix Reordering Algorithms
di: Tang, Tao, et al.
Pubblicazione: (2025)

LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
di: Gu, Diandian, et al.
Pubblicazione: (2024)

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient
di: Deng, Xiaoge, et al.
Pubblicazione: (2021)

torch-sla: Differentiable Sparse Linear Algebra with Adjoint Solvers and Sparse Tensor Parallelism for PyTorch
di: Chi, Mingyuan, et al.
Pubblicazione: (2026)

A Hierarchical Sharded Blockchain Balancing Performance and Availability
di: Jo, Yongrae, et al.
Pubblicazione: (2025)

Dynamically Sharded Ledgers on a Distributed Hash Table
di: Fink, Christoffer, et al.
Pubblicazione: (2024)

Stable Blockchain Sharding under Adversarial Transaction Generation
di: Adhikari, Ramesh, et al.
Pubblicazione: (2024)

Investigating Sharding Advancements, Methodologies, and Adoption Potential in Hedera
di: Wang, Ziwei, et al.
Pubblicazione: (2025)

SparseMap: Loop Mapping for Sparse CNNs on Streaming Coarse-grained Reconfigurable Array
di: Ni, Xiaobing, et al.
Pubblicazione: (2024)

Fully-Distributed Byzantine Agreement in Sparse Networks
di: Augustine, John, et al.
Pubblicazione: (2024)

cuFastTuckerPlus: A Stochastic Parallel Sparse FastTucker Decomposition Using GPU Tensor Cores
di: Li, Zixuan, et al.
Pubblicazione: (2024)

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
di: Pan, Xinglin, et al.
Pubblicazione: (2024)

Near-Optimal Stability for Distributed Transaction Processing in Blockchain Sharding
di: Adhikari, Ramesh, et al.
Pubblicazione: (2025)

SmartShards: Churn-Tolerant Continuously Available Distributed Ledger
di: Oglio, Joseph, et al.
Pubblicazione: (2025)