Cover Image

ProxyAttn: Guided Sparse Attention via Representative Heads

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Yixuan, He, Huang, Bao, Siqi, Wu, Hua, Wang, Haifeng, Zhu, Qingfu, Che, Wanxiang
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Multi-Layer Attention is the Amplifier of Demonstration Effectiveness
by: Wang, Dingzirui, et al.
Published: (2025)

CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMs
by: Xu, Yuzhuang, et al.
Published: (2024)

CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis
by: Xu, Yuzhuang, et al.
Published: (2025)

SpecAttn: Speculating Sparse Attention
by: Shah, Harsh
Published: (2025)

Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training
by: Wang, Yixuan, et al.
Published: (2024)

AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache
by: Song, Dinghong, et al.
Published: (2025)

CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing
by: Wang, Yixuan, et al.
Published: (2025)

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
by: Ji, Shiyu, et al.
Published: (2026)

Improving Grammatical Error Correction via Contextual Data Augmentation
by: Wang, Yixuan, et al.
Published: (2024)

Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models
by: Sun, Shaoning, et al.
Published: (2026)

Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding
by: Wang, Yixuan, et al.
Published: (2025)

Seer Self-Consistency: Advance Budget Estimation for Adaptive Test-Time Scaling
by: Ji, Shiyu, et al.
Published: (2025)

In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks
by: Wang, Dingzirui, et al.
Published: (2024)

Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
by: Filipek, Adam
Published: (2025)

Scaling Linear Attention with Sparse State Expansion
by: Pan, Yuqi, et al.
Published: (2025)

Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling
by: Luo, Xianzhen, et al.
Published: (2024)

Model Unlearning via Sparse Autoencoder Subspace Guided Projections
by: Wang, Xu, et al.
Published: (2025)

Scaling Laws for Agent Harnesses via Effective Feedback Compute
by: Zhang, Xuanliang, et al.
Published: (2026)

SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
by: Wang, Hanrui, et al.
Published: (2020)

ComplexFormer: Disruptively Advancing Transformer Inference Ability via Head-Specific Complex Vector Attention
by: Shao, Jintian, et al.
Published: (2025)

Humanizing the Machine: Proxy Attacks to Mislead LLM Detectors
by: Wang, Tianchun, et al.
Published: (2024)

InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration
by: Wang, Fali, et al.
Published: (2024)

Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
by: Piękos, Piotr, et al.
Published: (2025)

AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation
by: Luo, Lvzhou, et al.
Published: (2025)

AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers
by: Achtibat, Reduan, et al.
Published: (2024)

Trainable Dynamic Mask Sparse Attention
by: Shi, Jingze, et al.
Published: (2025)

Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query
by: Wang, Yixuan, et al.
Published: (2025)

ProxySPEX: Inference-Efficient Interpretability via Sparse Feature Interactions in LLMs
by: Butler, Landon, et al.
Published: (2025)

Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning
by: Wang, Junxuan, et al.
Published: (2025)

SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models
by: He, Jinghan, et al.
Published: (2024)

Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
by: He, Mutian, et al.
Published: (2025)

DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
by: Chen, Yilong, et al.
Published: (2024)

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
by: He, Zhengfu, et al.
Published: (2025)

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
by: Nawrot, Piotr, et al.
Published: (2025)

PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention
by: Chen, Lida, et al.
Published: (2025)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
by: Yuan, Jingyang, et al.
Published: (2025)

NOSA: Native and Offloadable Sparse Attention
by: Huang, Yuxiang, et al.
Published: (2025)

CHAI: Clustered Head Attention for Efficient LLM Inference
by: Agarwal, Saurabh, et al.
Published: (2024)

Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification
by: Li, Bohan, et al.
Published: (2023)

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
by: Fu, Zizhuo, et al.
Published: (2026)