Imagen de Portada

Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Fujii, Kazuki, Watanabe, Kohei, Yokota, Rio
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning Distributed, Parallel, and Cluster Computing
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
por: Wang, Yujie, et al.
Publicado: (2024)

Heterogeneous Parallelism for Multimodal Large Language Model Training
por: Karnati, Yashaswi, et al.
Publicado: (2026)

DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction
por: Zhang, Yanqi, et al.
Publicado: (2024)

Efficient Parallelization Layouts for Large-Scale Distributed Model Training
por: Hagemann, Johannes, et al.
Publicado: (2023)

AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
por: Chen, Ling, et al.
Publicado: (2026)

BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Training
por: Wu, Houming, et al.
Publicado: (2024)

Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training
por: Zhang, Xin, et al.
Publicado: (2025)

MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
por: Liu, Dennis, et al.
Publicado: (2025)

Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference
por: Luo, Shuqing, et al.
Publicado: (2025)

Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design
por: Xue, Chunyu, et al.
Publicado: (2024)

Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment
por: Li, Haoyang, et al.
Publicado: (2024)

EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
por: Chen, Yanxi, et al.
Publicado: (2023)

Lion Cub: Minimizing Communication Overhead in Distributed Lion
por: Ishikawa, Satoki, et al.
Publicado: (2024)

FedPM: Federated Learning Using Second-order Optimization with Preconditioned Mixing of Local Parameters
por: Ishii, Hiro, et al.
Publicado: (2025)

Improving Automatic Parallel Training via Balanced Memory Workload Optimization
por: Wang, Yujie, et al.
Publicado: (2023)

GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelism
por: Polisetty, Sandeep, et al.
Publicado: (2023)

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
por: Wu, Bingyang, et al.
Publicado: (2024)

ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
por: Wang, Yujia, et al.
Publicado: (2025)

TawPipe: Topology-Aware Weight Pipeline Parallelism for Accelerating Long-Context Large Models Training
por: Wu, Houming, et al.
Publicado: (2025)

TurboGR: An Accelerated Training System for Large-Scale Generative Recommendation
por: Chai, Huichao, et al.
Publicado: (2026)

DHO$_2$: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM
por: Gu, Shunxian, et al.
Publicado: (2025)

SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
por: Jia, Jinda, et al.
Publicado: (2024)

Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performance
por: Piao, XinYu, et al.
Publicado: (2021)

DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
por: Wang, Yuanqing, et al.
Publicado: (2026)

Go With The Flow: Churn-Tolerant Decentralized Training of Large Language Models
por: Blagoev, Nikolay, et al.
Publicado: (2025)

Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networks
por: Waleffe, Roger, et al.
Publicado: (2025)

Optimization of Energy Consumption Forecasting in Puno using Parallel Computing and ARIMA Models: An Innovative Approach to Big Data Processing
por: Vilca-Tinta, Cliver W., et al.
Publicado: (2024)

HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
por: Zhang, Geng, et al.
Publicado: (2025)

PaSE: Parallelization Strategies for Efficient DNN Training
por: Elango, Venmugil
Publicado: (2024)

Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
por: Ghadia, Ravi, et al.
Publicado: (2026)

Memory and Bandwidth are All You Need for Fully Sharded Data Parallel
por: Wang, Jiangtao, et al.
Publicado: (2025)

NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining
por: Jiang, Zhida, et al.
Publicado: (2026)

BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
por: Wang, Zhengyang, et al.
Publicado: (2025)

DHP: Efficient Scaling of MLLM Training with Dynamic Hybrid Parallelism
por: Niu, Yifan, et al.
Publicado: (2026)

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
por: Jiang, Ziheng, et al.
Publicado: (2024)

Pipette: Automatic Fine-grained Large Language Model Training Configurator for Real-World Clusters
por: Yim, Jinkyu, et al.
Publicado: (2024)

Enhancing Memory Efficiency in Large Language Model Training Through Chronos-aware Pipeline Parallelism
por: Lin, Xinyuan, et al.
Publicado: (2025)

Reducing Energy Bloat in Large Model Training
por: Chung, Jae-Won, et al.
Publicado: (2023)

On Optimizing the Communication of Model Parallelism
por: Zhuang, Yonghao, et al.
Publicado: (2022)

Pipeline Parallelism with Controllable Memory
por: Qi, Penghui, et al.
Publicado: (2024)