Image de couverture de livre

Quantifying LLM Attention-Head Stability: Implications for Circuit Universality

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Bali, Karan, Stanley, Jack, Suresh, Praneet, Bzdok, Danilo
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers
par: Suresh, Praneet, et autres
Publié: (2025)

Steering CLIP's vision transformer with sparse autoencoders
par: Joseph, Sonia, et autres
Publié: (2025)

Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Video
par: Joseph, Sonia, et autres
Publié: (2025)

Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
par: Xu, Yongzhong
Publié: (2026)

The Uncanny Valley: A Comprehensive Analysis of Diffusion Models
par: Ghanem, Karam, et autres
Publié: (2024)

Can Active Label Correction Improve LLM-based Modular AI Systems?
par: Taneja, Karan, et autres
Publié: (2024)

To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability
par: Lee, Joonhyung, et autres
Publié: (2024)

Singular Vectors of Attention Heads Align with Features
par: Franco, Gabriel, et autres
Publié: (2026)

ThinkTank: A Framework for Generalizing Domain-Specific AI Agent Systems into Universal Collaborative Intelligence Platforms
par: Surabhi, Praneet Sai Madhu, et autres
Publié: (2025)

Neuronal Attention Circuit (NAC) for Representation Learning
par: Razzaq, Waleed, et autres
Publié: (2025)

Stock Market Price Prediction: A Hybrid LSTM and Sequential Self-Attention based Approach
par: Pardeshi, Karan, et autres
Publié: (2023)

HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
par: Luo, Cheng, et autres
Publié: (2025)

Geometric Analysis of Token Selection in Multi-Head Attention
par: Mudarisov, Timur, et autres
Publié: (2026)

Tractable Sharpness-Aware Learning of Probabilistic Circuits
par: Suresh, Hrithik, et autres
Publié: (2025)

Do Attention Heads Compete or Cooperate during Counting?
par: Zsámboki, Pál, et autres
Publié: (2025)

Superiority of Multi-Head Attention in In-Context Linear Regression
par: Cui, Yingqian, et autres
Publié: (2024)

The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms
par: Otsuka, Hikari, et autres
Publié: (2025)

Estimating Unknown Population Sizes Using the Hypergeometric Distribution
par: Hodgson, Liam, et autres
Publié: (2024)

Circuit Complexity of Hierarchical Knowledge Tracing and Implications for Log-Precision Transformers
par: Liu, Naiming, et autres
Publié: (2026)

Neuronal Stochastic Attention Circuit (NSAC) for Probabilistic Representation Learning
par: Razzaq, Waleed, et autres
Publié: (2026)

TransMLA: Multi-Head Latent Attention Is All You Need
par: Meng, Fanxu, et autres
Publié: (2025)

Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient
par: Wang, George, et autres
Publié: (2024)

MoH: Multi-Head Attention as Mixture-of-Head Attention
par: Jin, Peng, et autres
Publié: (2024)

Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systems
par: Ross, Brendan Leigh, et autres
Publié: (2025)

Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
par: Li, Cheng, et autres
Publié: (2025)

On the Existence of Universal Simulators of Attention
par: Dutta, Debanjan, et autres
Publié: (2025)

Universal Approximation with Softmax Attention
par: Hu, Jerry Yao-Chieh, et autres
Publié: (2025)

Exact Attention Sensitivity and the Geometry of Transformer Stability
par: Emadi, Seyed Morteza
Publié: (2026)

Which Attention Heads Matter for In-Context Learning?
par: Yin, Kayo, et autres
Publié: (2025)

How Much Thinking is Enough? Quantifying and Understanding Redundancy in LLM Reasoning
par: Zhai, Zhiyuan, et autres
Publié: (2026)

CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention
par: Zhou, Zhongzhu, et autres
Publié: (2026)

Fourier-Mixed Window Attention: Accelerating Informer for Long Sequence Time-Series Forecasting
par: Tran, Nhat Thanh, et autres
Publié: (2023)

Even Heads Fix Odd Errors: Mechanistic Discovery and Surgical Repair in Transformer Attention
par: Sandoval, Gustavo
Publié: (2025)

SafeSeek: Universal Attribution of Safety Circuits in Language Models
par: Yu, Miao, et autres
Publié: (2026)

Sparse Attention Decomposition Applied to Circuit Tracing
par: Franco, Gabriel, et autres
Publié: (2024)

Towards LLM-guided Efficient and Interpretable Multi-linear Tensor Network Rank Selection
par: Iacovides, Giorgos, et autres
Publié: (2024)

Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment
par: Chen, Tiejin, et autres
Publié: (2026)

When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
par: Xu, Yongzhong
Publié: (2026)

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
par: Urrutia, Felipe, et autres
Publié: (2026)

How Many Heads Make an SSM? A Unified Framework for Attention and State Space Models
par: Ghodsi, Ali
Publié: (2025)