Cover Image

Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Luo, Cheng, Zhao, Jiawei, Chen, Zhuoming, Chen, Beidi, Anandkumar, Anima
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models
by: Zhang, Junyang, et al.
Published: (2025)

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
by: Zhao, Jiawei, et al.
Published: (2024)

InRank: Incremental Low-Rank Learning
by: Zhao, Jiawei, et al.
Published: (2023)

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
by: Sun, Hanshi, et al.
Published: (2024)

HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
by: Luo, Cheng, et al.
Published: (2025)

TensorGRaD: Tensor Gradient Robust Decomposition for Memory-Efficient Neural Operator Training
by: Loeschcke, Sebastian, et al.
Published: (2025)

Fast Training of Diffusion Models with Masked Transformers
by: Zheng, Hongkai, et al.
Published: (2023)

StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence Training of LLMs
by: Luo, Qijun, et al.
Published: (2025)

Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs
by: George, Robert Joseph, et al.
Published: (2022)

Calibrated Uncertainty Quantification for Operator Learning via Conformal Prediction
by: Ma, Ziqi, et al.
Published: (2024)

Fourier Neural Operators Explained: A Practical Perspective
by: Duruisseaux, Valentin, et al.
Published: (2025)

Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
by: Zheng, Haizhong, et al.
Published: (2025)

EcoSpa: Efficient Transformer Training with Coupled Sparsity
by: Xiao, Jinqi, et al.
Published: (2025)

Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences
by: Bekman, Stas, et al.
Published: (2025)

HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
by: Zhang, Geng, et al.
Published: (2025)

MonarchRT: Efficient Attention for Real-Time Video Generation
by: Agarwal, Krish, et al.
Published: (2026)

Kinetics: Rethinking Test-Time Scaling Laws
by: Sadhukhan, Ranajoy, et al.
Published: (2025)

Mechanistic Interpretability with Sparse Autoencoder Neural Operators
by: Tolooshams, Bahareh, et al.
Published: (2025)

SMR: State Memory Replay for Long Sequence Modeling
by: Qi, Biqing, et al.
Published: (2024)

Solving Poisson Equations using Neural Walk-on-Spheres
by: Nam, Hong Chul, et al.
Published: (2024)

Geometric Operator Learning with Optimal Transport
by: Li, Xinyi, et al.
Published: (2025)

Lean Copilot: Large Language Models as Copilots for Theorem Proving in Lean
by: Song, Peiyang, et al.
Published: (2024)

LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory
by: Liu, Zicheng, et al.
Published: (2024)

FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel
by: Yan, Ran, et al.
Published: (2025)

Diffusion State-Guided Projected Gradient for Inverse Problems
by: Zirvi, Rayhan, et al.
Published: (2024)

Sequence Complementor: Complementing Transformers For Time Series Forecasting with Learnable Sequences
by: Chen, Xiwen, et al.
Published: (2025)

DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training
by: Hao, Zhongkai, et al.
Published: (2024)

AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference
by: Zhao, Xuanlei, et al.
Published: (2024)

IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs
by: Mao, Yuzhen, et al.
Published: (2024)

DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
by: Zhao, Xuanlei, et al.
Published: (2024)

VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training
by: Shen, Guobin, et al.
Published: (2026)

Flatten Graphs as Sequences: Transformers are Scalable Graph Generators
by: Chen, Dexiong, et al.
Published: (2025)

Fourier Neural Operator with Learned Deformations for PDEs on General Geometries
by: Li, Zongyi, et al.
Published: (2022)

Sequential Controlled Langevin Diffusions
by: Chen, Junhua, et al.
Published: (2024)

Memory by Design: Probabilistic Sequence Layers
by: Dowling, Matthew, et al.
Published: (2026)

Introduction to Sequence Modeling with Transformers
by: Kämäräinen, Joni-Kristian
Published: (2025)

Transforming Chatbot Text: A Sequence-to-Sequence Approach
by: Reddy, Natesh, et al.
Published: (2025)

Transformers Simulate MLE for Sequence Generation in Bayesian Networks
by: Cao, Yuan, et al.
Published: (2025)

Enhancing Transformer-based models for Long Sequence Time Series Forecasting via Structured Matrix
by: Zhang, Zhicheng, et al.
Published: (2024)

On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Era
by: Tiezzi, Matteo, et al.
Published: (2024)