Cover Image

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Sok, Jaewon, Yeom, Jewon, Park, Seonghyeon, Park, Jeongjae, Kim, Taesup
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

EpiCaR: Knowing What You Don't Know Matters for Better Reasoning in LLMs
by: Yeom, Jewon, et al.
Published: (2026)

Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer
by: Yeom, Jewon, et al.
Published: (2026)

Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillation
by: Park, Seonghyeon, et al.
Published: (2026)

From Noise to Diversity: Random Embedding Injection in LLM Reasoning
by: Kim, Heejun, et al.
Published: (2026)

Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization
by: Son, Seungwoo, et al.
Published: (2024)

CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
by: Li, Zixuan, et al.
Published: (2025)

Sink-Aware Pruning for Diffusion Language Models
by: Myrzakhan, Aidar, et al.
Published: (2026)

Attention Sinks in Diffusion Language Models
by: Rulli, Maximo Eduardo, et al.
Published: (2025)

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
by: Choi, Jiho, et al.
Published: (2026)

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
by: Fu, Zizhuo, et al.
Published: (2026)

To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
by: Luo, Jiayun, et al.
Published: (2025)

Efficient Streaming Language Models with Attention Sinks
by: Xiao, Guangxuan, et al.
Published: (2023)

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
by: Sun, Shangwen, et al.
Published: (2026)

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
by: Qiu, Zihan, et al.
Published: (2025)

Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
by: Binkowski, Jakub, et al.
Published: (2026)

How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
by: Peng, Runyu, et al.
Published: (2026)

When Attention Sink Emerges in Language Models: An Empirical View
by: Gu, Xiangming, et al.
Published: (2024)

On the Existence and Behavior of Secondary Attention Sinks
by: Wong, Jeffrey T. H., et al.
Published: (2025)

Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration
by: Yu, Zhongzhi, et al.
Published: (2024)

Spectral Filters, Dark Signals, and Attention Sinks
by: Cancedda, Nicola
Published: (2024)

SinkTrack: Attention Sink based Context Anchoring for Large Language Models
by: Liu, Xu, et al.
Published: (2026)

SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Models
by: Zhang, Hengyu
Published: (2024)

ASAP: Attention Sink Anchored Pruning
by: Lee, Jaehyuk, et al.
Published: (2026)

Beyond Case Law: Evaluating Structure-Aware Retrieval and Safety in Statute-Centric Legal QA
by: Chae, Kyubyung, et al.
Published: (2026)

KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
by: Su, Zunhai, et al.
Published: (2025)

Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning
by: Shang, Bingqi, et al.
Published: (2025)

Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
by: Zhang, Stephen, et al.
Published: (2025)

"Well, Keep Thinking": Enhancing LLM Reasoning with Adaptive Injection Decoding
by: Jin, Hyunbin, et al.
Published: (2025)

Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation
by: Mutisya, Hillary, et al.
Published: (2026)

Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
by: Meng, Tian, et al.
Published: (2024)

One Token Is Enough: Improving Diffusion Language Models with a Sink Token
by: Zhang, Zihou, et al.
Published: (2026)

A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
by: Qiu, Zihan, et al.
Published: (2026)

A Little Leak Will Sink a Great Ship: Survey of Transparency for Large Language Models from Start to Finish
by: Kaneko, Masahiro, et al.
Published: (2024)

Prompt-based Depth Pruning of Large Language Models
by: Wee, Juyun, et al.
Published: (2025)

PRISP: Privacy-Safe Few-Shot Personalization via Lightweight Adaptation
by: Park, Junho, et al.
Published: (2026)

Zero-Shot RTL Code Generation with Attention Sink Augmented Large Language Models
by: Sandal, Selim, et al.
Published: (2024)

Does RoBERTa Perform Better than BERT in Continual Learning: An Attention Sink Perspective
by: Bai, Xueying, et al.
Published: (2024)

When LLM Therapists Become Salespeople: Evaluating Large Language Models for Ethical Motivational Interviewing
by: Kong, Haein, et al.
Published: (2025)

See What You Are Told: Visual Attention Sink in Large Multimodal Models
by: Kang, Seil, et al.
Published: (2025)

Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling
by: Grand, Gabriel, et al.
Published: (2024)