Cover Image

Rectified Sparse Attention

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Sun, Yutao, Ye, Tianzhu, Dong, Li, Xia, Yuqing, Chen, Jian, Gao, Yizhao, Cao, Shijie, Wang, Jianyong, Wei, Furu
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Universal YOCO for Efficient Depth Scaling
by: Sun, Yutao, et al.
Published: (2026)

Differential Transformer
by: Ye, Tianzhu, et al.
Published: (2024)

Reinforcement Pre-Training
by: Dong, Qingxiu, et al.
Published: (2025)

SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
by: Gao, Yizhao, et al.
Published: (2024)

On-Policy Context Distillation for Language Models
by: Ye, Tianzhu, et al.
Published: (2026)

Online Experiential Learning for Language Models
by: Ye, Tianzhu, et al.
Published: (2026)

You Only Cache Once: Decoder-Decoder Architectures for Language Models
by: Sun, Yutao, et al.
Published: (2024)

SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
by: Gao, Yizhao, et al.
Published: (2025)

HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
by: Gao, Yizhao, et al.
Published: (2026)

Multimodal Latent Language Modeling with Next-Token Diffusion
by: Sun, Yutao, et al.
Published: (2024)

Black-Box On-Policy Distillation of Large Language Models
by: Ye, Tianzhu, et al.
Published: (2025)

Efficient Attention Mechanisms for Large Language Models: A Survey
by: Sun, Yutao, et al.
Published: (2025)

Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
by: Wang, Hongyu, et al.
Published: (2024)

Maximum Score Routing For Mixture-of-Experts
by: Dong, Bowen, et al.
Published: (2025)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
by: Yuan, Jingyang, et al.
Published: (2025)

PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention
by: Chen, Lida, et al.
Published: (2025)

SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
by: Shen, Zhenyi, et al.
Published: (2025)

Mixture-of-Depths Attention
by: Zhu, Lianghui, et al.
Published: (2026)

Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention
by: Yang, Lijie, et al.
Published: (2025)

Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
by: Zhang, Di, et al.
Published: (2026)

Bitnet.cpp: Efficient Edge Inference for Ternary LLMs
by: Wang, Jinheng, et al.
Published: (2025)

MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
by: Li, Wenxuan, et al.
Published: (2025)

SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
by: Mou, Yutao, et al.
Published: (2024)

MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognition
by: Xia, Yinfeng, et al.
Published: (2025)

Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping
by: Chen, Yao, et al.
Published: (2026)

Semi-Parametric Retrieval via Binary Bag-of-Tokens Index
by: Zhou, Jiawei, et al.
Published: (2024)

mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data
by: Chen, Haonan, et al.
Published: (2025)

SparseD: Sparse Attention for Diffusion Language Models
by: Wang, Zeqing, et al.
Published: (2025)

Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
by: Wu, Wenshan, et al.
Published: (2024)

Optimizing Prompts for Text-to-Image Generation
by: Hao, Yaru, et al.
Published: (2022)

A Unified Sparse Attention via Multi-Granularity Compression
by: Liu, Siran, et al.
Published: (2025)

Direct Preference Knowledge Distillation for Large Language Models
by: Li, Yixing, et al.
Published: (2024)

MiniLLM: On-Policy Distillation of Large Language Models
by: Gu, Yuxian, et al.
Published: (2023)

Little Giants: Synthesizing High-Quality Embedding Data at Scale
by: Chen, Haonan, et al.
Published: (2024)

BitNet Distillation
by: Wu, Xun, et al.
Published: (2025)

Information-Preserving Reformulation of Reasoning Traces for Antidistillation
by: Ding, Jiayu, et al.
Published: (2025)

Exact Hard Monotonic Attention for Character-Level Transduction
by: Wu, Shijie, et al.
Published: (2019)

Towards Optimal Learning of Language Models
by: Gu, Yuxian, et al.
Published: (2024)

Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
by: Hu, Yuxuan, et al.
Published: (2025)

Thinking Augmented Pre-training
by: Wang, Liang, et al.
Published: (2025)