Copertina

Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Yu, Zhongzhi, Wang, Zheng, Fu, Yonggan, Shi, Huihong, Shaikh, Khalid, Lin, Yingyan Celine
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
di: Fu, Yonggan, et al.
Pubblicazione: (2020)

When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
di: You, Haoran, et al.
Pubblicazione: (2024)

MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generation
di: Zhang, Yongan, et al.
Pubblicazione: (2024)

ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks
di: You, Haoran, et al.
Pubblicazione: (2022)

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
di: Qiu, Zihan, et al.
Pubblicazione: (2025)

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
di: Shi, Dachuan, et al.
Pubblicazione: (2025)

A3C-S: Automated Agent Accelerator Co-Search towards Efficient Deep Reinforcement Learning
di: Fu, Yonggan, et al.
Pubblicazione: (2021)

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
di: Fu, Zizhuo, et al.
Pubblicazione: (2026)

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
di: Sok, Jaewon, et al.
Pubblicazione: (2026)

Attention Sinks in Diffusion Language Models
di: Rulli, Maximo Eduardo, et al.
Pubblicazione: (2025)

ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
di: You, Haoran, et al.
Pubblicazione: (2024)

AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment
di: Fu, Yonggan, et al.
Pubblicazione: (2024)

GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models
di: Fu, Yonggan, et al.
Pubblicazione: (2023)

LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
di: Ye, Zhifan, et al.
Pubblicazione: (2025)

CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
di: Li, Zixuan, et al.
Pubblicazione: (2025)

Efficient Streaming Language Models with Attention Sinks
di: Xiao, Guangxuan, et al.
Pubblicazione: (2023)

Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
di: Binkowski, Jakub, et al.
Pubblicazione: (2026)

How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
di: Peng, Runyu, et al.
Pubblicazione: (2026)

When Attention Sink Emerges in Language Models: An Empirical View
di: Gu, Xiangming, et al.
Pubblicazione: (2024)

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
di: Sun, Shangwen, et al.
Pubblicazione: (2026)

A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
di: Qiu, Zihan, et al.
Pubblicazione: (2026)

Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition
di: Duan, Yifei, et al.
Pubblicazione: (2025)

Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization
di: Son, Seungwoo, et al.
Pubblicazione: (2024)

Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields
di: Fu, Yonggan, et al.
Pubblicazione: (2024)

Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing
di: Fu, Yonggan, et al.
Pubblicazione: (2022)

KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
di: Su, Zunhai, et al.
Pubblicazione: (2025)

On the Existence and Behavior of Secondary Attention Sinks
di: Wong, Jeffrey T. H., et al.
Pubblicazione: (2025)

Sliding Window Attention Training for Efficient Large Language Models
di: Fu, Zichuan, et al.
Pubblicazione: (2025)

Spectral Filters, Dark Signals, and Attention Sinks
di: Cancedda, Nicola
Pubblicazione: (2024)

ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer
di: You, Haoran, et al.
Pubblicazione: (2023)

Early-Bird GCNs: Graph-Network Co-Optimization Towards More Efficient GCN Training and Inference via Drawing Early-Bird Lottery Tickets
di: You, Haoran, et al.
Pubblicazione: (2021)

$R^2$-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction
di: Du, Zhenbang, et al.
Pubblicazione: (2026)

Zero-Shot RTL Code Generation with Attention Sink Augmented Large Language Models
di: Sandal, Selim, et al.
Pubblicazione: (2024)

2-in-1 Accelerator: Enabling Random Precision Switch for Winning Both Adversarial Robustness and Efficiency
di: Fu, Yonggan, et al.
Pubblicazione: (2021)

Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning
di: Chen, Zhongzhi, et al.
Pubblicazione: (2023)

Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Models
di: Fu, Yu, et al.
Pubblicazione: (2025)

Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
di: Meng, Tian, et al.
Pubblicazione: (2024)

Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models
di: Yu, Zeping, et al.
Pubblicazione: (2025)

HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark
di: Li, Chaojian, et al.
Pubblicazione: (2021)

Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
di: Zhang, Stephen, et al.
Pubblicazione: (2025)