Cover Image

Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Qin, Zhen, Sun, Weigao, Li, Dong, Shen, Xuyang, Sun, Weixuan, Zhong, Yiran
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
by: Qin, Zhen, et al.
Published: (2024)

HGRN2: Gated Linear RNNs with State Expansion
by: Qin, Zhen, et al.
Published: (2024)

Scaling Laws for Linear Complexity Language Models
by: Shen, Xuyang, et al.
Published: (2024)

CO2: Efficient Distributed Training with Full Communication-Computation Overlap
by: Sun, Weigao, et al.
Published: (2024)

Linear Attention Sequence Parallelism
by: Sun, Weigao, et al.
Published: (2024)

TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
by: Qin, Zhen, et al.
Published: (2023)

Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective
by: Qin, Zhen, et al.
Published: (2024)

Elucidating the Design Space of Decay in Linear Attention
by: Qin, Zhen, et al.
Published: (2025)

You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet
by: Qin, Zhen, et al.
Published: (2024)

LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid
by: Sun, Weigao, et al.
Published: (2025)

MiniMax-01: Scaling Foundation Models with Lightning Attention
by: MiniMax, et al.
Published: (2025)

FlashSampling: Fast and Memory-Efficient Exact Sampling
by: Ruiz, Tomas, et al.
Published: (2026)

Native Hybrid Attention for Efficient Sequence Modeling
by: Du, Jusen, et al.
Published: (2025)

LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training
by: Qu, Xiaoye, et al.
Published: (2024)

Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme
by: Ma, Yan, et al.
Published: (2025)

MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
by: MiniMax, et al.
Published: (2025)

Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
by: Sun, Weigao, et al.
Published: (2025)

Efficient Attention Mechanisms for Large Language Models: A Survey
by: Sun, Yutao, et al.
Published: (2025)

Liger: Linearizing Large Language Models to Gated Recurrent Structures
by: Lan, Disen, et al.
Published: (2025)

In-game Toxic Language Detection: Shared Task and Attention Residuals
by: Jia, Yuanzhe, et al.
Published: (2022)

Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling
by: Hu, Xiang, et al.
Published: (2024)

CogniBench: A Legal-inspired Framework and Dataset for Assessing Cognitive Faithfulness of Large Language Models
by: Tang, Xiaqiang, et al.
Published: (2025)

In-Context Former: Lightning-fast Compressing Context for Large Language Model
by: Wang, Xiangfeng, et al.
Published: (2024)

Length-MAX Tokenizer for Language Models
by: Dong, Dong, et al.
Published: (2025)

On Speeding Up Language Model Evaluation
by: Zhou, Jin Peng, et al.
Published: (2024)

MoM: Linear Sequence Modeling with Mixture-of-Memories
by: Du, Jusen, et al.
Published: (2025)

DAPE V2: Process Attention Score as Feature Map for Length Extrapolation
by: Zheng, Chuanyang, et al.
Published: (2024)

Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention
by: Wang, Weixuan, et al.
Published: (2024)

AALC: Large Language Model Efficient Reasoning via Adaptive Accuracy-Length Control
by: Li, Ruosen, et al.
Published: (2025)

A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond
by: Qu, Xiaoye, et al.
Published: (2025)

Efficient Pretraining Length Scaling
by: Wu, Bohong, et al.
Published: (2025)

Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
by: Sun, Weigao, et al.
Published: (2025)

Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey
by: Qin, Weicong, et al.
Published: (2024)

Liaozhai through the Looking-Glass: On Paratextual Explicitation of Culture-Bound Terms in Machine Translation
by: Shen, Sherrie, et al.
Published: (2025)

HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Models
by: Wang, Weixuan, et al.
Published: (2025)

SentenceVAE: Enable Next-sentence Prediction for Large Language Models with Faster Speed, Higher Accuracy and Longer Context
by: An, Hongjun, et al.
Published: (2024)

Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
by: Zhang, Zhen, et al.
Published: (2026)

Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models
by: Wu, Qiong, et al.
Published: (2024)

Correlation-Aware Select and Merge Attention for Efficient Fine-Tuning and Context Length Extension
by: Wang, Ning, et al.
Published: (2024)

LIFEBench: Evaluating Length Instruction Following in Large Language Models
by: Zhang, Wei, et al.
Published: (2025)