Copertina

SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Kim, Han-Byul, Hoang, Duc, Kundu, Arnav, Samragh, Mohammad, Cho, Minsik
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation
di: Cho, Minsik, et al.
Pubblicazione: (2024)

TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training
di: Liu, Man, et al.
Pubblicazione: (2026)

Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization
di: Zhu, Zhanda, et al.
Pubblicazione: (2025)

TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelism
di: Cho, Seonghye, et al.
Pubblicazione: (2026)

Can Large Language Models Write Parallel Code?
di: Nichols, Daniel, et al.
Pubblicazione: (2024)

Research on Model Parallelism and Data Parallelism Optimization Methods in Large Language Model-Based Recommendation Systems
di: Yang, Haowei, et al.
Pubblicazione: (2025)

Oases: Efficient Large-Scale Model Training on Commodity Servers via Overlapped and Automated Tensor Model Parallelism
di: Li, Shengwei, et al.
Pubblicazione: (2023)

HPC-Coder: Modeling Parallel Programs using Large Language Models
di: Nichols, Daniel, et al.
Pubblicazione: (2023)

ACE-Sync: An Adaptive Cloud-Edge Synchronization Framework for Communication-Efficient Large-Scale Distributed Model Training
di: Yang, Yi, et al.
Pubblicazione: (2025)

Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training
di: Sun, Ao, et al.
Pubblicazione: (2024)

Synergistic Tensor and Pipeline Parallelism
di: Qi, Mengshi, et al.
Pubblicazione: (2025)

Can Large Language Models Predict Parallel Code Performance?
di: Bolet, Gregory, et al.
Pubblicazione: (2025)

Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
di: Zhao, Long, et al.
Pubblicazione: (2026)

SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learning
di: Yang, Yuning, et al.
Pubblicazione: (2024)

Efficient Multi-Model Orchestration for Self-Hosted Large Language Models
di: Vangala, Bhanu Prakash, et al.
Pubblicazione: (2025)

torch-sla: Differentiable Sparse Linear Algebra with Adjoint Solvers and Sparse Tensor Parallelism for PyTorch
di: Chi, Mingyuan, et al.
Pubblicazione: (2026)

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
di: Chen, Qiaoling, et al.
Pubblicazione: (2024)

TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networks
di: Shi, Ziji, et al.
Pubblicazione: (2023)

Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures
di: Georganas, Evangelos, et al.
Pubblicazione: (2023)

ZeroPP: Unleashing Exceptional Parallelism Efficiency through Tensor-Parallelism-Free Methodology
di: Tang, Ding, et al.
Pubblicazione: (2024)

InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training
di: Wang, Shiju, et al.
Pubblicazione: (2025)

FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
di: Gao, Shouwei, et al.
Pubblicazione: (2026)

CFP: Efficient Optimization of Intra-Operator Parallelism Plans for Large Model Training
di: Hu, Weifang, et al.
Pubblicazione: (2025)

Amoeba: Runtime Tensor Parallel Transformation for LLM Inference Services
di: Chen, Haoyu, et al.
Pubblicazione: (2025)

Accelerating Heterogeneous Tensor Parallelism via Flexible Workload Control
di: Wang, Zhigang, et al.
Pubblicazione: (2024)

Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
di: Srivatsa, Vikranth, et al.
Pubblicazione: (2026)

EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models
di: Cheng, Jialiang, et al.
Pubblicazione: (2024)

Efficient Parallel Compilation and Profiling of Quantum Circuits at Large Scales
di: Moore, Jane, et al.
Pubblicazione: (2026)

COPUS: Co-adaptive Parallelism and Batch Size Selection in Large Language Model Training
di: Sakip, Akhmed, et al.
Pubblicazione: (2026)

Minimizing Communication for Parallel Symmetric Tensor Times Same Vector Computation
di: Daas, Hussam Al, et al.
Pubblicazione: (2025)

Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
di: Pan, Xinglin, et al.
Pubblicazione: (2025)

Astra: Efficient and Money-saving Automatic Parallel Strategies Search on Heterogeneous GPUs
di: Wang, Peiran, et al.
Pubblicazione: (2025)

SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
di: Xie, Jincheng, et al.
Pubblicazione: (2026)

DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
di: Huang, You-Liang, et al.
Pubblicazione: (2026)

TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training
di: Han, Shujie, et al.
Pubblicazione: (2026)

Enhancing Memory Efficiency in Large Language Model Training Through Chronos-aware Pipeline Parallelism
di: Lin, Xinyuan, et al.
Pubblicazione: (2025)

AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism
di: Xu, Wendong, et al.
Pubblicazione: (2025)

Scaling Performance of Large Language Model Pretraining
di: Interrante-Grant, Alexander, et al.
Pubblicazione: (2025)

Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization
di: Li, Haoyang, et al.
Pubblicazione: (2024)

Opara: Exploiting Operator Parallelism for Expediting DNN Inference on GPUs
di: Chen, Aodong, et al.
Pubblicazione: (2023)