Image de couverture de livre

BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Sun, Ao, Zhao, Weilin, Han, Xu, Yang, Cheng, Liu, Zhiyuan, Shi, Chuan, Sun, Maosong
Format:	Preprint
Publié:	2024
Sujets:	Distributed, Parallel, and Cluster Computing Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens
par: Sun, Ao, et autres
Publié: (2025)

Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training
par: Sun, Ao, et autres
Publié: (2024)

HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
par: Zhang, Geng, et autres
Publié: (2025)

AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
par: Liu, Di, et autres
Publié: (2026)

Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
par: Lin, Bin, et autres
Publié: (2024)

LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
par: Gu, Diandian, et autres
Publié: (2024)

Zen-Attention: A Compiler Framework for Dynamic Attention Folding on AMD NPUs
par: Deshmukh, Aadesh, et autres
Publié: (2025)

SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading
par: Chen, Qiaoling, et autres
Publié: (2025)

SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
par: Zhou, Qihui, et autres
Publié: (2025)

Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism
par: Li, Cong, et autres
Publié: (2025)

Mesh-Attention: A New Communication-Efficient Distributed Attention with Improved Data Locality
par: Chen, Sirui, et autres
Publié: (2025)

StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training
par: Liu, Ziming, et autres
Publié: (2024)

Janus: Disaggregating Attention and Experts for Scalable MoE Inference
par: Zhang, Zhexiang, et autres
Publié: (2025)

Bandwidth-Aware and Cost-Efficient Pipeline Parallel Scheduling in Geo-Distributed LLM Training
par: Zhang, Han, et autres
Publié: (2026)

Joint Temporal-Structural Representation Learning for Distributed Fault Discrimination in Microservice Architectures
par: Xue, Yihan, et autres
Publié: (2026)

S-HPLB: Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance
par: Liu, Di, et autres
Publié: (2026)

DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training
par: Wang, Zhixin, et autres
Publié: (2025)

MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
par: Li, Wenxuan, et autres
Publié: (2025)

InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training
par: Wang, Shiju, et autres
Publié: (2025)

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
par: Wu, Bingyang, et autres
Publié: (2024)

MegatronApp: Efficient and Comprehensive Management on Distributed LLM Training
par: Zhao, Bohan, et autres
Publié: (2025)

MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference
par: Rhee, Myunghyun, et autres
Publié: (2025)

Efficient Long-context Language Model Training by Core Attention Disaggregation
par: Zhuang, Yonghao, et autres
Publié: (2025)

MoLink: Distributed and Efficient Serving Framework for Large Models
par: Jin, Lewei, et autres
Publié: (2025)

Opt-GPTQ: An Optimized GPTQ Combining Sparse Attention and Quantization Techniques
par: Kong, Jie, et autres
Publié: (2025)

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient
par: Deng, Xiaoge, et autres
Publié: (2021)

Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking
par: Mo, Zizhao, et autres
Publié: (2026)

AGAThA: Fast and Efficient GPU Acceleration of Guided Sequence Alignment for Long Read Mapping
par: Park, Seongyeon, et autres
Publié: (2024)

PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel
par: Yi, Jinjun, et autres
Publié: (2025)

AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference
par: Zhao, Xuanlei, et autres
Publié: (2024)

MTGenRec: An Efficient Distributed Training System for Generative Recommendation Models in Meituan
par: Wang, Yuxiang, et autres
Publié: (2025)

SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learning
par: Yang, Yuning, et autres
Publié: (2024)

Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters
par: Zhang, WenZheng, et autres
Publié: (2024)

Akita: A High Usability Simulation Framework for Computer Architecture
par: Jannat, Sabila Al, et autres
Publié: (2026)

FlowWalker: A Memory-efficient and High-performance GPU-based Dynamic Graph Random Walk Framework
par: Mei, Junyi, et autres
Publié: (2024)

Hestia: Hyperthread-Level Scheduling for Cloud Microservices with Interference-Aware Attention
par: Yang, Dingyu, et autres
Publié: (2026)

PATCHEDSERVE: A Patch Management Framework for SLO-Optimized Hybrid Resolution Diffusion Serving
par: Sun, Desen, et autres
Publié: (2025)

Is Flash Attention Stable?
par: Golden, Alicia, et autres
Publié: (2024)

Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems
par: Liu, Guowei, et autres
Publié: (2026)

From Attention to Disaggregation: Tracing the Evolution of LLM Inference
par: Kumar, Madabattula Rajesh, et autres
Publié: (2025)