Imagen de Portada

Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Yu, Zhongzhi, Wang, Zheng, Fu, Yonggan, Shi, Huihong, Shaikh, Khalid, Lin, Yingyan Celine
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
por: Fu, Yonggan, et al.
Publicado: (2020)

When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
por: You, Haoran, et al.
Publicado: (2024)

MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generation
por: Zhang, Yongan, et al.
Publicado: (2024)

ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks
por: You, Haoran, et al.
Publicado: (2022)

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
por: Qiu, Zihan, et al.
Publicado: (2025)

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
por: Shi, Dachuan, et al.
Publicado: (2025)

A3C-S: Automated Agent Accelerator Co-Search towards Efficient Deep Reinforcement Learning
por: Fu, Yonggan, et al.
Publicado: (2021)

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
por: Fu, Zizhuo, et al.
Publicado: (2026)

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
por: Sok, Jaewon, et al.
Publicado: (2026)

Attention Sinks in Diffusion Language Models
por: Rulli, Maximo Eduardo, et al.
Publicado: (2025)

ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
por: You, Haoran, et al.
Publicado: (2024)

AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment
por: Fu, Yonggan, et al.
Publicado: (2024)

GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models
por: Fu, Yonggan, et al.
Publicado: (2023)

LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
por: Ye, Zhifan, et al.
Publicado: (2025)

CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
por: Li, Zixuan, et al.
Publicado: (2025)

Efficient Streaming Language Models with Attention Sinks
por: Xiao, Guangxuan, et al.
Publicado: (2023)

Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
por: Binkowski, Jakub, et al.
Publicado: (2026)

How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
por: Peng, Runyu, et al.
Publicado: (2026)

When Attention Sink Emerges in Language Models: An Empirical View
por: Gu, Xiangming, et al.
Publicado: (2024)

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
por: Sun, Shangwen, et al.
Publicado: (2026)

A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
por: Qiu, Zihan, et al.
Publicado: (2026)

Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition
por: Duan, Yifei, et al.
Publicado: (2025)

Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization
por: Son, Seungwoo, et al.
Publicado: (2024)

Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields
por: Fu, Yonggan, et al.
Publicado: (2024)

Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing
por: Fu, Yonggan, et al.
Publicado: (2022)

KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
por: Su, Zunhai, et al.
Publicado: (2025)

On the Existence and Behavior of Secondary Attention Sinks
por: Wong, Jeffrey T. H., et al.
Publicado: (2025)

Sliding Window Attention Training for Efficient Large Language Models
por: Fu, Zichuan, et al.
Publicado: (2025)

Spectral Filters, Dark Signals, and Attention Sinks
por: Cancedda, Nicola
Publicado: (2024)

ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer
por: You, Haoran, et al.
Publicado: (2023)

Early-Bird GCNs: Graph-Network Co-Optimization Towards More Efficient GCN Training and Inference via Drawing Early-Bird Lottery Tickets
por: You, Haoran, et al.
Publicado: (2021)

$R^2$-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction
por: Du, Zhenbang, et al.
Publicado: (2026)

Zero-Shot RTL Code Generation with Attention Sink Augmented Large Language Models
por: Sandal, Selim, et al.
Publicado: (2024)

2-in-1 Accelerator: Enabling Random Precision Switch for Winning Both Adversarial Robustness and Efficiency
por: Fu, Yonggan, et al.
Publicado: (2021)

Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning
por: Chen, Zhongzhi, et al.
Publicado: (2023)

Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Models
por: Fu, Yu, et al.
Publicado: (2025)

Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
por: Meng, Tian, et al.
Publicado: (2024)

Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models
por: Yu, Zeping, et al.
Publicado: (2025)

HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark
por: Li, Chaojian, et al.
Publicado: (2021)

Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
por: Zhang, Stephen, et al.
Publicado: (2025)