Image de couverture de livre

Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Lou, Chao, Jia, Zixia, Zheng, Zilong, Tu, Kewei
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Sparser, Faster, Lighter Transformer Language Models
par: Cetin, Edoardo, et autres
Publié: (2026)

Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
par: Zhuang, Xialie, et autres
Publié: (2025)

Adaptive Preference Optimization with Uncertainty-aware Utility Anchor
par: Wang, Xiaobo, et autres
Publié: (2025)

AdaSplash-2: Faster Differentiable Sparse Attention
par: Gonçalves, Nuno, et autres
Publié: (2026)

Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage
par: Hu, Junhao, et autres
Publié: (2026)

Supernova: Achieving More with Less in Transformer Architectures
par: Tanase, Andrei-Valentin, et autres
Publié: (2025)

Faster Transformer Decoding: N-gram Masked Self-Attention
par: Chelba, Ciprian, et autres
Publié: (2020)

Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs
par: Liu, Jinzhe, et autres
Publié: (2025)

Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning
par: Schulte, David, et autres
Publié: (2024)

Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection
par: Jo, Dongwon, et autres
Publié: (2026)

Should We Attend More or Less? Modulating Attention for Fairness
par: Zayed, Abdelrahman, et autres
Publié: (2023)

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
par: Nawrot, Piotr, et autres
Publié: (2025)

Pay Less Attention to Function Words for Free Robustness of Vision-Language Models
par: Tian, Qiwei, et autres
Publié: (2025)

FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference
par: Lai, Xunhao, et autres
Publié: (2025)

Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models
par: Zhao, Yida, et autres
Publié: (2024)

The NLP Task Effectiveness of Long-Range Transformers
par: Qin, Guanghui, et autres
Publié: (2022)

Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
par: Noukhovitch, Michael, et autres
Publié: (2024)

MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
par: MiniCPM Team, et autres
Publié: (2026)

STS: Efficient Sparse Attention with Speculative Token Sparsity
par: Xu, Ceyu, et autres
Publié: (2026)

SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
par: Han, Jiayi, et autres
Publié: (2024)

LIMR: Less is More for RL Scaling
par: Li, Xuefeng, et autres
Publié: (2025)

Less is More: Denoising Knowledge Graphs For Retrieval Augmented Generation
par: Zheng, Yilun, et autres
Publié: (2025)

SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention
par: Zhu, Qianchao, et autres
Publié: (2024)

MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
par: Li, Wenxuan, et autres
Publié: (2025)

Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access
par: Hu, Xiang, et autres
Publié: (2025)

From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency
par: Wen, Kaiyue, et autres
Publié: (2024)

Two-stage LLM Fine-tuning with Less Specialization and More Generalization
par: Wang, Yihan, et autres
Publié: (2022)

Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM
par: Fan, Zehao, et autres
Publié: (2025)

Less is More: Improving LLM Alignment via Preference Data Selection
par: Deng, Xun, et autres
Publié: (2025)

Star Attention: Efficient LLM Inference over Long Sequences
par: Acharya, Shantanu, et autres
Publié: (2024)

Sparser Block-Sparse Attention via Token Permutation
par: Wang, Xinghao, et autres
Publié: (2025)

Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs
par: Zhang, Yifei, et autres
Publié: (2024)

Multipole Attention for Efficient Long Context Reasoning
par: Hooper, Coleman, et autres
Publié: (2025)

Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
par: Filipek, Adam
Publié: (2025)

Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning
par: Wang, Junxuan, et autres
Publié: (2025)

BiSparse-AAS: Bilinear Sparse Attention and Adaptive Spans Framework for Scalable and Efficient Text Summarization
par: Hagos, Desta Haileselassie, et autres
Publié: (2025)

Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts
par: Ye, Jiayuan, et autres
Publié: (2026)

Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning
par: Shrivastava, Vaishnavi, et autres
Publié: (2025)

Scaling Linear Attention with Sparse State Expansion
par: Pan, Yuqi, et autres
Publié: (2025)

Explain Less, Understand More: Jargon Detection via Personalized Parameter-Efficient Fine-tuning
par: Wu, Bohao, et autres
Publié: (2025)