Imagen de Portada

Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Bekman, Stas, Rajbhandari, Samyam, Wyatt, Michael, Rasley, Jeff, Ruwase, Tunji, Yao, Zhewei, Qiao, Aurick, He, Yuxiong
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads
por: Hidayetoglu, Mert, et al.
Publicado: (2025)

SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation
por: Qiao, Aurick, et al.
Publicado: (2024)

Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI
por: Rajbhandari, Samyam, et al.
Publicado: (2025)

MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
por: Su, Zhaoyuan, et al.
Publicado: (2026)

STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning
por: Lee, Jaeseong, et al.
Publicado: (2024)

OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs
por: Lee, Jaeseong, et al.
Publicado: (2025)

Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Parallelis
por: Lian, Xinyu, et al.
Publicado: (2024)

DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
por: Holmes, Connor, et al.
Publicado: (2024)

AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
por: Gupta, Ahan, et al.
Publicado: (2026)

Federated Timeline Synthesis: Scalable and Private Methodology For Model Training and Deployment
por: Renc, Pawel, et al.
Publicado: (2025)

DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
por: Li, Conglong, et al.
Publicado: (2022)

FastPersist: Accelerating Model Checkpointing in Deep Learning
por: Wang, Guanhua, et al.
Publicado: (2024)

Fast and Accurate Causal Parallel Decoding using Jacobi Forcing
por: Hu, Lanxiang, et al.
Publicado: (2025)

FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design
por: Xia, Haojun, et al.
Publicado: (2024)

Training a Large Language Model for Medical Coding Using Privacy-Preserving Synthetic Clinical Data
por: Cook, John, et al.
Publicado: (2026)

LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
por: Gu, Diandian, et al.
Publicado: (2024)

ExCoT: Optimizing Reasoning for Text-to-SQL with Execution Feedback
por: Zhai, Bohan, et al.
Publicado: (2025)

BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens
por: Sun, Ao, et al.
Publicado: (2025)

Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Training
por: Luo, Cheng, et al.
Publicado: (2024)

MedOrch: Medical Diagnosis with Tool-Augmented Reasoning Agents for Flexible Extensibility
por: He, Yexiao, et al.
Publicado: (2025)

Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
por: Chen, Fahao, et al.
Publicado: (2024)

Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer
por: Yao, Jinghan, et al.
Publicado: (2024)

StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence Training of LLMs
por: Luo, Qijun, et al.
Publicado: (2025)

Learning to Hint for Reinforcement Learning
por: Xia, Yu, et al.
Publicado: (2026)

R$^3$-SQL: Ranking Reward and Resampling for Text-to-SQL
por: Han, Hojae, et al.
Publicado: (2026)

Training-free LLM-generated Text Detection by Mining Token Probability Sequences
por: Xu, Yihuai, et al.
Publicado: (2024)

Pretext Training Algorithms for Event Sequence Data
por: Wang, Yimu, et al.
Publicado: (2024)

Context Parallelism for Scalable Million-Token Inference
por: Yang, Amy, et al.
Publicado: (2024)

SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading
por: Chen, Qiaoling, et al.
Publicado: (2025)

HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
por: Zhang, Geng, et al.
Publicado: (2025)

Multi-word Tokenization for Sequence Compression
por: Gee, Leonidas, et al.
Publicado: (2024)

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
por: Zhang, Zhenyu, et al.
Publicado: (2024)

Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
por: Li, Wenhao, et al.
Publicado: (2026)

SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips
por: Lian, Xinyu, et al.
Publicado: (2025)

Learning to Self-Evolve
por: Chen, Xiaoyin, et al.
Publicado: (2026)

Token Sequence Compression for Efficient Multimodal Computing
por: Omri, Yasmine, et al.
Publicado: (2025)

Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens
por: Erba, Vittorio, et al.
Publicado: (2024)

Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale
por: Guo, Liang, et al.
Publicado: (2026)

Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
por: Shyam, Vasu, et al.
Publicado: (2026)

Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQL
por: Yao, Zhewei, et al.
Publicado: (2025)