Image de couverture de livre

Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Li, Zhexiang, Wang, Haoyu, Bao, Yutong, Woodruff, David
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

LevAttention: Time, Space, and Streaming Efficient Algorithm for Heavy Attentions
par: Kannan, Ravindran, et autres
Publié: (2024)

Ridge Leverage Score Sampling for $\ell_p$ Subspace Approximation
par: Woodruff, David P., et autres
Publié: (2024)

LOOKAT: Lookup-Optimized Key-Attention for Memory-Efficient Transformers
par: Karmore, Aryan
Publié: (2026)

Stronger Graph Transformer with Regularized Attention Scores
par: Ku, Eugene
Publié: (2023)

Causal Information Prioritization for Efficient Reinforcement Learning
par: Cao, Hongye, et autres
Publié: (2025)

Mixture-of-Channels: Exploiting Sparse FFNs for Efficient LLMs Pre-Training and Inference
par: Wu, Tong, et autres
Publié: (2025)

Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge
par: Wang, Penghao, et autres
Publié: (2025)

QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models
par: Wang, Yutong, et autres
Publié: (2025)

Simulation-based Inference via Langevin Dynamics with Score Matching
par: Jiang, Haoyu, et autres
Publié: (2025)

Hamming Attention Distillation: Binarizing Keys and Queries for Efficient Long-Context Transformers
par: Horton, Mark, et autres
Publié: (2025)

John Ellipsoids via Lazy Updates
par: Woodruff, David P., et autres
Publié: (2025)

Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients
par: Muhamed, Aashiq, et autres
Publié: (2024)

EcoSpa: Efficient Transformer Training with Coupled Sparsity
par: Xiao, Jinqi, et autres
Publié: (2025)

Loki: Low-rank Keys for Efficient Sparse Attention
par: Singhania, Prajwal, et autres
Publié: (2024)

The Effect of Attention Head Count on Transformer Approximation
par: Yu, Penghao, et autres
Publié: (2025)

Unmasking Backdoors: An Explainable Defense via Gradient-Attention Anomaly Scoring for Pre-trained Language Models
par: Das, Anindya Sundar, et autres
Publié: (2025)

Even Sparser Graph Transformers
par: Shirzad, Hamed, et autres
Publié: (2024)

Query-Efficient Locally Private Hypothesis Selection via the Scheffe Graph
par: Kamath, Gautam, et autres
Publié: (2025)

Learning to Focus: Prioritizing Informative Histories with Structured Attention Mechanisms in Partially Observable Reinforcement Learning
par: Allegue, Daniel De Dios, et autres
Publié: (2025)

A Theory for Compressibility of Graph Transformers for Transductive Learning
par: Shirzad, Hamed, et autres
Publié: (2024)

ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
par: Zhang, Qiuyang, et autres
Publié: (2026)

Toward Efficient Testing of Graph Neural Networks via Test Input Prioritization
par: Yang, Lichen, et autres
Publié: (2025)

GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values
par: Javadi, Farnoosh, et autres
Publié: (2023)

Prioritized Generative Replay
par: Wang, Renhao, et autres
Publié: (2024)

Brenier Isotonic Regression
par: Bao, Han, et autres
Publié: (2026)

BitStopper: An Efficient Transformer Attention Accelerator via Stage-fusion and Early Termination
par: Wang, Huizheng, et autres
Publié: (2025)

Graph Attention Based Prioritization of Disease Responsible Genes from Multimodal Alzheimer's Network
par: Teji, Binon, et autres
Publié: (2026)

Attention-Enhanced Prioritized Proximal Policy Optimization for Adaptive Edge Caching
par: Niknia, Farnaz, et autres
Publié: (2024)

AnchorGT: Efficient and Flexible Attention Architecture for Scalable Graph Transformers
par: Zhu, Wenhao, et autres
Publié: (2024)

First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Training
par: Kim, Gyudong, et autres
Publié: (2025)

Solving Attention Kernel Regression Problem via Pre-conditioner
par: Song, Zhao, et autres
Publié: (2023)

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
par: Brandon, William, et autres
Publié: (2024)

MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
par: Liu, Dong, et autres
Publié: (2026)

Attention-Only Transformers via Unrolled Subspace Denoising
par: Wang, Peng, et autres
Publié: (2025)

Exploring Quantization for Efficient Pre-Training of Transformer Language Models
par: Chitsaz, Kamran, et autres
Publié: (2024)

Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model
par: Jin, Jiarui, et autres
Publié: (2025)

Provably Efficient Algorithm for Best Scoring Rule Identification in Online Principal-Agent Information Acquisition
par: Wang, Zichen, et autres
Publié: (2025)

RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers
par: Liu, Yuxi, et autres
Publié: (2026)

Gated Linear Attention Transformers with Hardware-Efficient Training
par: Yang, Songlin, et autres
Publié: (2023)

Exploring Key Factors for Long-Term Vessel Incident Risk Prediction
par: Chen, Tianyi, et autres
Publié: (2024)