Copertina

Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Su, Zunhai, Zhang, Hengyuan, Wu, Wei, Zhang, Yifan, Liu, Yaxiu, Xiao, He, Yang, Qingyao, Sun, Yuxuan, Yang, Rui, Zhang, Chao, Fan, Keyu, Ye, Weihao, Xiong, Jing, Shen, Hui, Tao, Chaofan, Wu, Taiqiang, Wan, Zhongwei, Qian, Yulei, Xie, Yuchen, Wong, Ngai
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
di: Wu, Taiqiang, et al.
Pubblicazione: (2024)

MMFormalizer: Multimodal Autoformalization in the Wild
di: Xiong, Jing, et al.
Pubblicazione: (2026)

XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression
di: Su, Zunhai, et al.
Pubblicazione: (2026)

XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression
di: Su, Zunhai, et al.
Pubblicazione: (2026)

KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
di: Su, Zunhai, et al.
Pubblicazione: (2025)

SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining
di: Zhang, Yifan, et al.
Pubblicazione: (2026)

CktFormalizer: Autoformalization of Natural Language into Circuit Representations
di: Xiong, Jing, et al.
Pubblicazione: (2026)

Revisiting Model Interpolation for Efficient Reasoning
di: Wu, Taiqiang, et al.
Pubblicazione: (2025)

The Art of Efficient Reasoning: Data, Reward, and Optimization
di: Wu, Taiqiang, et al.
Pubblicazione: (2026)

Mixture-of-Subspaces in Low-Rank Adaptation
di: Wu, Taiqiang, et al.
Pubblicazione: (2024)

Unveiling Super Experts in Mixture-of-Experts Large Language Models
di: Su, Zunhai, et al.
Pubblicazione: (2025)

BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models
di: Chen, Junyu, et al.
Pubblicazione: (2026)

Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity
di: Zhang, Hengyuan, et al.
Pubblicazione: (2026)

Enhancing Robustness of Implicit Neural Representations Against Weight Perturbations
di: Zhou, Wenyong, et al.
Pubblicazione: (2025)

MINR: Efficient Implicit Neural Representations for Multi-Image Encoding
di: Zhou, Wenyong, et al.
Pubblicazione: (2025)

Timber: Training-free Instruct Model Refining with Base via Effective Rank
di: Wu, Taiqiang, et al.
Pubblicazione: (2025)

DoPE: Denoising Rotary Position Embedding
di: Xiong, Jing, et al.
Pubblicazione: (2025)

LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
di: Yang, Runming, et al.
Pubblicazione: (2024)

Shadow-FT: Tuning Instruct Model via Training on Paired Base Model
di: Wu, Taiqiang, et al.
Pubblicazione: (2025)

Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation
di: Zhang, Hengyuan, et al.
Pubblicazione: (2025)

Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models
di: Xiao, He, et al.
Pubblicazione: (2025)

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models
di: Yang, Yifan, et al.
Pubblicazione: (2024)

Weight-Inherited Distillation for Task-Agnostic BERT Compression
di: Wu, Taiqiang, et al.
Pubblicazione: (2023)

Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs
di: Wu, Taiqiang, et al.
Pubblicazione: (2023)

CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
di: Li, Zixuan, et al.
Pubblicazione: (2025)

UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective
di: Xiong, Jing, et al.
Pubblicazione: (2024)

On the Existence and Behavior of Secondary Attention Sinks
di: Wong, Jeffrey T. H., et al.
Pubblicazione: (2025)

Distribution-Aware Hadamard Quantization for Hardware-Efficient Implicit Neural Representations
di: Zhou, Wenyong, et al.
Pubblicazione: (2025)

Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance
di: Chen, Xinrong, et al.
Pubblicazione: (2026)

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
di: Tao, Chaofan, et al.
Pubblicazione: (2024)

Autoregressive Models in Vision: A Survey
di: Xiong, Jing, et al.
Pubblicazione: (2024)

PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
di: Shen, Hui, et al.
Pubblicazione: (2025)

LoCa: Logit Calibration for Knowledge Distillation
di: Yang, Runming, et al.
Pubblicazione: (2024)

Can We Trust LLMs on Memristors? Diving into Reasoning Ability under Non-Ideality
di: Wu, Taiqiang, et al.
Pubblicazione: (2026)

MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers
di: Yang, Zebin, et al.
Pubblicazione: (2024)

FocusVLA: Focused Visual Utilization for Vision-Language-Action Models
di: Zhang, Yichi, et al.
Pubblicazione: (2026)

Attention Sinks and Outliers in Attention Residuals
di: Luo, Haozheng, et al.
Pubblicazione: (2026)

Re-Activating Frozen Primitives for 3D Gaussian Splatting
di: Cheng, Yuxin, et al.
Pubblicazione: (2025)

QuadINR: Hardware-Efficient Implicit Neural Representations Through Quadratic Activation
di: Zhou, Wenyong, et al.
Pubblicazione: (2025)

SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving
di: Xu, Wendong, et al.
Pubblicazione: (2025)