Imagen de Portada

Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Ghadia, Ravi, Abraham, Maksim, Vorobyov, Sergei, Ryabinin, Max
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning Distributed, Parallel, and Cluster Computing
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference
por: Zhao, Xuanlei, et al.
Publicado: (2024)

DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism
por: Jiang, Chenyu, et al.
Publicado: (2025)

ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inference
por: Meng, Han, et al.
Publicado: (2026)

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
por: Wu, Bingyang, et al.
Publicado: (2024)

Memory and Bandwidth are All You Need for Fully Sharded Data Parallel
por: Wang, Jiangtao, et al.
Publicado: (2025)

Improving Automatic Parallel Training via Balanced Memory Workload Optimization
por: Wang, Yujie, et al.
Publicado: (2023)

HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
por: Zhang, Geng, et al.
Publicado: (2025)

Efficient Long Context Fine-tuning with Chunk Flow
por: Yuan, Xiulong, et al.
Publicado: (2025)

Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design
por: Xue, Chunyu, et al.
Publicado: (2024)

DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction
por: Zhang, Yanqi, et al.
Publicado: (2024)

Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator
por: Fujii, Kazuki, et al.
Publicado: (2024)

PaSE: Parallelization Strategies for Efficient DNN Training
por: Elango, Venmugil
Publicado: (2024)

Pipeline Parallelism with Controllable Memory
por: Qi, Penghui, et al.
Publicado: (2024)

DHP: Efficient Scaling of MLLM Training with Dynamic Hybrid Parallelism
por: Niu, Yifan, et al.
Publicado: (2026)

Efficient Parallelization Layouts for Large-Scale Distributed Model Training
por: Hagemann, Johannes, et al.
Publicado: (2023)

Efficient Parallel Reinforcement Learning Framework using the Reactor Model
por: Kwok, Jacky, et al.
Publicado: (2023)

Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inference
por: Prabhakar, Rohan Baskar, et al.
Publicado: (2024)

Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification
por: Huang, Guang, et al.
Publicado: (2026)

MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
por: Liu, Dennis, et al.
Publicado: (2025)

ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
por: Liu, Zedong, et al.
Publicado: (2025)

SAIR: Cost-Efficient Multi-Stage ML Pipeline Autoscaling via In-Context Reinforcement Learning
por: Su, Jianchang, et al.
Publicado: (2026)

Scalable and Cost-Efficient ML Inference: Parallel Batch Processing with Serverless Functions
por: Barrak, Amine, et al.
Publicado: (2025)

CoCoDiff: Optimizing Collective Communications for Distributed Diffusion Transformer Inference Under Ulysses Sequence Parallelism
por: Ma, Bin, et al.
Publicado: (2026)

PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving
por: Bai, Xu, et al.
Publicado: (2026)

Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI
por: Rajbhandari, Samyam, et al.
Publicado: (2025)

Re-evaluating the Memory-balanced Pipeline Parallelism: BPipe
por: Huang, Mincong, et al.
Publicado: (2024)

DHO$_2$: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM
por: Gu, Shunxian, et al.
Publicado: (2025)

FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
por: Wang, Yujie, et al.
Publicado: (2024)

GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelism
por: Polisetty, Sandeep, et al.
Publicado: (2023)

Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networks
por: Waleffe, Roger, et al.
Publicado: (2025)

Vanishing Variance Problem in Fully Decentralized Neural-Network Systems
por: Tian, Yongding, et al.
Publicado: (2024)

On Optimizing the Communication of Model Parallelism
por: Zhuang, Yonghao, et al.
Publicado: (2022)

Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment
por: Li, Haoyang, et al.
Publicado: (2024)

Edge-Parallel Graph Encoder Embedding
por: Lubonja, Ariel, et al.
Publicado: (2024)

TASP: Topology-aware Sequence Parallelism
por: Wang, Yida, et al.
Publicado: (2025)

Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Training
por: Wu, Yebo, et al.
Publicado: (2024)

Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting
por: Tian, Chunlin, et al.
Publicado: (2024)

Context Parallelism for Scalable Million-Token Inference
por: Yang, Amy, et al.
Publicado: (2024)

FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission
por: Zhang, Zeling, et al.
Publicado: (2024)

High-Dimensional Sparse Data Low-rank Representation via Accelerated Asynchronous Parallel Stochastic Gradient Descent
por: Hu, Qicong, et al.
Publicado: (2024)