Imagen de Portada

Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Amer, Hossam, Karim, Rezaul, Pourranjbar, Ali, Zhang, Weiwei, Ahmed, Walid, Chen, Boxing
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Distributed, Parallel, and Cluster Computing
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training
por: Karim, Rezaul, et al.
Publicado: (2026)

Comparative Study of Large Language Model Architectures on Frontier
por: Yin, Junqi, et al.
Publicado: (2024)

MixServe: An Automatic Distributed Serving System for MoE Models with Hybrid Parallelism Based on Fused Communication Algorithm
por: Zhou, Bowen, et al.
Publicado: (2026)

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
por: Chen, Qiaoling, et al.
Publicado: (2024)

Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization
por: Li, Haoyang, et al.
Publicado: (2024)

ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training
por: Kang, Xueze, et al.
Publicado: (2025)

A Parallel and Distributed Rust Library for Core Decomposition on Large Graphs
por: Rucci, Davide, et al.
Publicado: (2025)

A Comparative Review of Parallel Exact, Heuristic, Metaheuristic, and Hybrid Optimization Techniques for the Traveling Salesman Problem
por: Alkhalifa, Rabab, et al.
Publicado: (2025)

FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
por: Gao, Shouwei, et al.
Publicado: (2026)

An Explorative Study on Distributed Computing Techniques in Training and Inference of Large Language Models
por: Hakim, Sheikh Azizul, et al.
Publicado: (2025)

The Entropy of Parallel Systems
por: Adefemi, Temitayo
Publicado: (2025)

gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling
por: Guo, Tianyu, et al.
Publicado: (2025)

COPUS: Co-adaptive Parallelism and Batch Size Selection in Large Language Model Training
por: Sakip, Akhmed, et al.
Publicado: (2026)

Hybrid-Parallel: Achieving High Performance and Energy Efficient Distributed Inference on Robots
por: Sun, Zekai, et al.
Publicado: (2024)

Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation
por: Wu, Tianyuan, et al.
Publicado: (2025)

HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference
por: Lin, Haoran, et al.
Publicado: (2025)

Cloud-native and Distributed Systems for Efficient and Scalable Large Language Models -- A Research Agenda
por: Xu, Minxian, et al.
Publicado: (2026)

6G Twin: Hybrid Gaussian Radio Fields for Channel Estimation and Non-Linear Precoder Design for Radio Access Networks
por: Mohsin, Muhammad Ahmed, et al.
Publicado: (2025)

Distributed Semi-Speculative Parallel Anisotropic Mesh Adaptation
por: Garner, Kevin, et al.
Publicado: (2026)

SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs
por: Yang, Jiacheng, et al.
Publicado: (2026)

Enhancing Memory Efficiency in Large Language Model Training Through Chronos-aware Pipeline Parallelism
por: Lin, Xinyuan, et al.
Publicado: (2025)

ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism
por: Ma, Tenghui, et al.
Publicado: (2026)

Adaptive Parallel Downloader for Large Genomic Datasets
por: Swargo, Rasman Mubtasim, et al.
Publicado: (2025)

Distributed And Parallel Low-Diameter Decompositions for Arbitrary and Restricted Graphs
por: Dou, Jinfeng, et al.
Publicado: (2024)

Parallel GPU-Enabled Algorithms for SpGEMM on Arbitrary Semirings with Hybrid Communication
por: McFarland, Thomas, et al.
Publicado: (2025)

High-Performance Sorting-Based k-mer Counting in Distributed Memory with Flexible Hybrid Parallelism
por: Li, Yifan, et al.
Publicado: (2024)

Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training
por: Sun, Ao, et al.
Publicado: (2024)

Efficient Data-Parallel Continual Learning with Asynchronous Distributed Rehearsal Buffers
por: Bouvier, Thomas, et al.
Publicado: (2024)

Characterizing Communication Patterns in Distributed Large Language Model Inference
por: Xu, Lang, et al.
Publicado: (2025)

PDSP-Bench: A Benchmarking System for Parallel and Distributed Stream Processing
por: Agnihotri, Pratyush, et al.
Publicado: (2025)

Automated Calibration of Parallel and Distributed Computing Simulators: A Case Study
por: McDonald, Jesse, et al.
Publicado: (2024)

Parallel Paradigms in Modern HPC: A Comparative Analysis of MPI, OpenMP, and CUDA
por: ALHafez, Nizar, et al.
Publicado: (2025)

HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving
por: Dong, Xianzhe, et al.
Publicado: (2025)

Edge System Design Using Containers and Unikernels for IoT Applications
por: Kaiser, Shahidullah, et al.
Publicado: (2024)

Efficient Parallel Compilation and Profiling of Quantum Circuits at Large Scales
por: Moore, Jane, et al.
Publicado: (2026)

Design and Implementation of Code Completion System Based on LLM and CodeBERT Hybrid Subsystem
por: Zhang, Bingbing, et al.
Publicado: (2025)

DRust: Language-Guided Distributed Shared Memory with Fine Granularity, Full Transparency, and Ultra Efficiency
por: Ma, Haoran, et al.
Publicado: (2024)

Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
por: Duan, Jiangfei, et al.
Publicado: (2024)

Cooling Matters: Benchmarking Large Language Models and Vision-Language Models on Liquid-Cooled Versus Air-Cooled H100 GPU Systems
por: Latif, Imran, et al.
Publicado: (2025)

A Novel Approach to Translate Structural Aggregation Queries to MapReduce Code
por: Abdelmoniem, Ahmed M., et al.
Publicado: (2025)