Copertina

Quantifying LLM Attention-Head Stability: Implications for Circuit Universality

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Bali, Karan, Stanley, Jack, Suresh, Praneet, Bzdok, Danilo
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers
di: Suresh, Praneet, et al.
Pubblicazione: (2025)

Steering CLIP's vision transformer with sparse autoencoders
di: Joseph, Sonia, et al.
Pubblicazione: (2025)

Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Video
di: Joseph, Sonia, et al.
Pubblicazione: (2025)

Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
di: Xu, Yongzhong
Pubblicazione: (2026)

The Uncanny Valley: A Comprehensive Analysis of Diffusion Models
di: Ghanem, Karam, et al.
Pubblicazione: (2024)

Can Active Label Correction Improve LLM-based Modular AI Systems?
di: Taneja, Karan, et al.
Pubblicazione: (2024)

To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability
di: Lee, Joonhyung, et al.
Pubblicazione: (2024)

Singular Vectors of Attention Heads Align with Features
di: Franco, Gabriel, et al.
Pubblicazione: (2026)

ThinkTank: A Framework for Generalizing Domain-Specific AI Agent Systems into Universal Collaborative Intelligence Platforms
di: Surabhi, Praneet Sai Madhu, et al.
Pubblicazione: (2025)

Neuronal Attention Circuit (NAC) for Representation Learning
di: Razzaq, Waleed, et al.
Pubblicazione: (2025)

Stock Market Price Prediction: A Hybrid LSTM and Sequential Self-Attention based Approach
di: Pardeshi, Karan, et al.
Pubblicazione: (2023)

HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
di: Luo, Cheng, et al.
Pubblicazione: (2025)

Geometric Analysis of Token Selection in Multi-Head Attention
di: Mudarisov, Timur, et al.
Pubblicazione: (2026)

Tractable Sharpness-Aware Learning of Probabilistic Circuits
di: Suresh, Hrithik, et al.
Pubblicazione: (2025)

Do Attention Heads Compete or Cooperate during Counting?
di: Zsámboki, Pál, et al.
Pubblicazione: (2025)

Superiority of Multi-Head Attention in In-Context Linear Regression
di: Cui, Yingqian, et al.
Pubblicazione: (2024)

The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms
di: Otsuka, Hikari, et al.
Pubblicazione: (2025)

Estimating Unknown Population Sizes Using the Hypergeometric Distribution
di: Hodgson, Liam, et al.
Pubblicazione: (2024)

Circuit Complexity of Hierarchical Knowledge Tracing and Implications for Log-Precision Transformers
di: Liu, Naiming, et al.
Pubblicazione: (2026)

Neuronal Stochastic Attention Circuit (NSAC) for Probabilistic Representation Learning
di: Razzaq, Waleed, et al.
Pubblicazione: (2026)

TransMLA: Multi-Head Latent Attention Is All You Need
di: Meng, Fanxu, et al.
Pubblicazione: (2025)

Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient
di: Wang, George, et al.
Pubblicazione: (2024)

MoH: Multi-Head Attention as Mixture-of-Head Attention
di: Jin, Peng, et al.
Pubblicazione: (2024)

Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systems
di: Ross, Brendan Leigh, et al.
Pubblicazione: (2025)

Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
di: Li, Cheng, et al.
Pubblicazione: (2025)

On the Existence of Universal Simulators of Attention
di: Dutta, Debanjan, et al.
Pubblicazione: (2025)

Universal Approximation with Softmax Attention
di: Hu, Jerry Yao-Chieh, et al.
Pubblicazione: (2025)

Exact Attention Sensitivity and the Geometry of Transformer Stability
di: Emadi, Seyed Morteza
Pubblicazione: (2026)

Which Attention Heads Matter for In-Context Learning?
di: Yin, Kayo, et al.
Pubblicazione: (2025)

How Much Thinking is Enough? Quantifying and Understanding Redundancy in LLM Reasoning
di: Zhai, Zhiyuan, et al.
Pubblicazione: (2026)

CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention
di: Zhou, Zhongzhu, et al.
Pubblicazione: (2026)

Fourier-Mixed Window Attention: Accelerating Informer for Long Sequence Time-Series Forecasting
di: Tran, Nhat Thanh, et al.
Pubblicazione: (2023)

Even Heads Fix Odd Errors: Mechanistic Discovery and Surgical Repair in Transformer Attention
di: Sandoval, Gustavo
Pubblicazione: (2025)

SafeSeek: Universal Attribution of Safety Circuits in Language Models
di: Yu, Miao, et al.
Pubblicazione: (2026)

Sparse Attention Decomposition Applied to Circuit Tracing
di: Franco, Gabriel, et al.
Pubblicazione: (2024)

Towards LLM-guided Efficient and Interpretable Multi-linear Tensor Network Rank Selection
di: Iacovides, Giorgos, et al.
Pubblicazione: (2024)

Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment
di: Chen, Tiejin, et al.
Pubblicazione: (2026)

When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
di: Xu, Yongzhong
Pubblicazione: (2026)

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
di: Urrutia, Felipe, et al.
Pubblicazione: (2026)

How Many Heads Make an SSM? A Unified Framework for Attention and State Space Models
di: Ghodsi, Ali
Pubblicazione: (2025)