Copertina

SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Jia, Jinda, Xie, Cong, Lu, Hanlin, Wang, Daoce, Feng, Hao, Zhang, Chengming, Sun, Baixi, Lin, Haibin, Zhang, Zhi, Liu, Xin, Tao, Dingwen
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Distributed, Parallel, and Cluster Computing
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data
di: Wang, Daoce, et al.
Pubblicazione: (2025)

Opt4GPTQ: Co-Optimizing Memory and Computation for 4-bit GPTQ Quantized LLM Inference on Heterogeneous Platforms
di: Zhang, Yaozheng, et al.
Pubblicazione: (2025)

DynaShard: Secure and Adaptive Blockchain Sharding Protocol with Hybrid Consensus and Dynamic Shard Management
di: Liu, Ao, et al.
Pubblicazione: (2024)

SP-Chain: Boosting Intra-Shard and Cross-Shard Security and Performance in Blockchain Sharding
di: Li, Mingzhe, et al.
Pubblicazione: (2024)

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
di: Chen, Qiaoling, et al.
Pubblicazione: (2024)

Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism
di: Qing, Yuhao, et al.
Pubblicazione: (2025)

TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training
di: Liu, Man, et al.
Pubblicazione: (2026)

Investigating Sharding Advancements, Methodologies, and Adoption Potential in Hedera
di: Wang, Ziwei, et al.
Pubblicazione: (2025)

StableShard: Stable and Scalable Blockchain Sharding with High Concurrency via Collaborative Committees
di: Li, Mingzhe, et al.
Pubblicazione: (2024)

EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
di: Zhang, Mingjin, et al.
Pubblicazione: (2024)

Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference
di: Yin, Ruokai, et al.
Pubblicazione: (2025)

TRAIL: Cross-Shard Validation for Cryptocurrency Byzantine Shard Protection
di: Jacovetty, Mitch, et al.
Pubblicazione: (2024)

A High-Quality Workflow for Multi-Resolution Scientific Data Reduction and Visualization
di: Wang, Daoce, et al.
Pubblicazione: (2024)

Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
di: Bhatia, Nidhi, et al.
Pubblicazione: (2025)

SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile
di: Zhang, Ruisi, et al.
Pubblicazione: (2024)

ShardTensor: Domain Parallelism for Scientific Machine Learning
di: Adams, Corey, et al.
Pubblicazione: (2026)

DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
di: Wang, Yuanqing, et al.
Pubblicazione: (2026)

Bandwidth-Aware and Cost-Efficient Pipeline Parallel Scheduling in Geo-Distributed LLM Training
di: Zhang, Han, et al.
Pubblicazione: (2026)

Fast Transaction Scheduling in Blockchain Sharding
di: Adhikari, Ramesh, et al.
Pubblicazione: (2024)

AI-driven Predictive Shard Allocation for Scalable Next Generation Blockchains
di: Haider, M. Zeeshan, et al.
Pubblicazione: (2025)

HiCoCS: High Concurrency Cross-Sharding on Permissioned Blockchains
di: Yang, Lingxiao, et al.
Pubblicazione: (2025)

Self-healing Nodes with Adaptive Data-Sharding
di: Thakur, Ayush, et al.
Pubblicazione: (2024)

On the Efficiency of Dynamic Transaction Scheduling in Blockchain Sharding
di: Adhikari, Ramesh, et al.
Pubblicazione: (2025)

Sharding Distributed Databases: A Critical Review
di: Solat, Siamak
Pubblicazione: (2024)

Memory and Bandwidth are All You Need for Fully Sharded Data Parallel
di: Wang, Jiangtao, et al.
Pubblicazione: (2025)

SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading
di: Chen, Qiaoling, et al.
Pubblicazione: (2025)

SpiralShard: Highly Concurrent and Secure Blockchain Sharding via Linked Cross-shard Endorsement
di: Lin, You, et al.
Pubblicazione: (2024)

ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
di: Liu, Zedong, et al.
Pubblicazione: (2025)

VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference
di: Liu, Zihan, et al.
Pubblicazione: (2025)

A Hierarchical Sharded Blockchain Balancing Performance and Availability
di: Jo, Yongrae, et al.
Pubblicazione: (2025)

Dynamically Sharded Ledgers on a Distributed Hash Table
di: Fink, Christoffer, et al.
Pubblicazione: (2024)

Stable Blockchain Sharding under Adversarial Transaction Generation
di: Adhikari, Ramesh, et al.
Pubblicazione: (2024)

ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism
di: Ma, Tenghui, et al.
Pubblicazione: (2026)

Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
di: Wang, Guanhua, et al.
Pubblicazione: (2024)

Near-Optimal Stability for Distributed Transaction Processing in Blockchain Sharding
di: Adhikari, Ramesh, et al.
Pubblicazione: (2025)

SmartShards: Churn-Tolerant Continuously Available Distributed Ledger
di: Oglio, Joseph, et al.
Pubblicazione: (2025)

MixServe: An Automatic Distributed Serving System for MoE Models with Hybrid Parallelism Based on Fused Communication Algorithm
di: Zhou, Bowen, et al.
Pubblicazione: (2026)

Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization
di: Li, Jinhao, et al.
Pubblicazione: (2023)

Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution
di: Wang, Haiquan, et al.
Pubblicazione: (2024)

AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training
di: Chen, Qiaoling, et al.
Pubblicazione: (2023)