Cover Image

Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Guo, Tianyu, Pai, Druv, Bai, Yu, Jiao, Jiantao, Jordan, Michael I., Mei, Song
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Toward a Theory of Tokenization in LLMs
by: Rajaraman, Nived, et al.
Published: (2024)

Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction
by: Wu, Ziyang, et al.
Published: (2024)

Attention-Only Transformers via Unrolled Subspace Denoising
by: Wang, Peng, et al.
Published: (2025)

In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention
by: He, Jianliang, et al.
Published: (2025)

Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers
by: Bu, Rui, et al.
Published: (2025)

Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers
by: Huang, Yixiao, et al.
Published: (2025)

Masked Completion via Structured Diffusion with White-Box Transformers
by: Pai, Druv, et al.
Published: (2024)

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
by: Zhu, Banghua, et al.
Published: (2024)

On the Edge of Memorization in Diffusion Models
by: Buchanan, Sam, et al.
Published: (2025)

Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons
by: Zhu, Banghua, et al.
Published: (2023)

GSM-Agent: Understanding Agentic Reasoning Using Controllable Environments
by: Zhu, Hanlin, et al.
Published: (2025)

Towards Anytime-Valid Statistical Watermarking
by: Huang, Baihe, et al.
Published: (2026)

A Global Geometric Analysis of Maximal Coding Rate Reduction
by: Wang, Peng, et al.
Published: (2024)

Demystifying the Slash Pattern in Attention: The Role of RoPE
by: Cheng, Yuan, et al.
Published: (2026)

Geometric Analysis of Token Selection in Multi-Head Attention
by: Mudarisov, Timur, et al.
Published: (2026)

Demystifying the Token Dynamics of Deep Selective State Space Models
by: Vo, Thieu N, et al.
Published: (2024)

Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations
by: Hu, Wentao, et al.
Published: (2026)

Attention Head Entropy of LLMs Predicts Answer Correctness
by: Ostmeier, Sophie, et al.
Published: (2026)

Efficient LLMs with AMP: Attention Heads and MLP Pruning
by: Mugnaini, Leandro Giusti, et al.
Published: (2025)

Collaborative Heterogeneous Causal Inference Beyond Meta-analysis
by: Guo, Tianyu, et al.
Published: (2024)

SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
by: Wang, Hanrui, et al.
Published: (2020)

Transformers Provably Learn to Internalize Chain-of-Thought
by: Huang, Yixiao, et al.
Published: (2026)

How Do LLMs Perform Two-Hop Reasoning in Context?
by: Guo, Tianyu, et al.
Published: (2025)

Even Heads Fix Odd Errors: Mechanistic Discovery and Surgical Repair in Transformer Attention
by: Sandoval, Gustavo
Published: (2025)

Multi-Head Low-Rank Attention
by: Liu, Songtao, et al.
Published: (2026)

Demystifying Prediction Powered Inference
by: Song, Yilin, et al.
Published: (2026)

Demystifying Long Chain-of-Thought Reasoning in LLMs
by: Yeo, Edward, et al.
Published: (2025)

HalluGuard: Demystifying Data-Driven and Reasoning-Driven Hallucinations in LLMs
by: Zeng, Xinyue, et al.
Published: (2026)

Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning
by: Su, DiJia, et al.
Published: (2025)

STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
by: Meng, Weikang, et al.
Published: (2026)

The Effect of Attention Head Count on Transformer Approximation
by: Yu, Penghao, et al.
Published: (2025)

Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics
by: Zhu, Hanlin, et al.
Published: (2024)

Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale
by: Rohweder, Jonas, et al.
Published: (2026)

Sample Complexity and Representation Ability of Test-time Scaling Paradigms
by: Huang, Baihe, et al.
Published: (2025)

Efficient Prompt Caching via Embedding Similarity
by: Zhu, Hanlin, et al.
Published: (2024)

Interleaved Head Attention
by: Duvvuri, Sai Surya, et al.
Published: (2026)

Demystifying Oversmoothing in Attention-Based Graph Neural Networks
by: Wu, Xinyi, et al.
Published: (2023)

Independent and Decentralized Learning in Markov Potential Games
by: Maheshwari, Chinmay, et al.
Published: (2022)

HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
by: Yang, Dongquan, et al.
Published: (2025)

Adaptive Head Budgeting for Efficient Multi-Head Attention
by: Faye, Bilal, et al.
Published: (2026)