Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs

Documenti analoghi

• Toward a Theory of Tokenization in LLMs
  di: Rajaraman, Nived, et al.
  Pubblicazione: (2024)
• Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction
  di: Wu, Ziyang, et al.
  Pubblicazione: (2024)
• Attention-Only Transformers via Unrolled Subspace Denoising
  di: Wang, Peng, et al.
  Pubblicazione: (2025)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention
  di: He, Jianliang, et al.
  Pubblicazione: (2025)
• Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers
  di: Bu, Rui, et al.
  Pubblicazione: (2025)