Copertina

Singular Vectors of Attention Heads Align with Features

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Franco, Gabriel, Loughridge, Carson, Crovella, Mark
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Sparse Attention Decomposition Applied to Circuit Tracing
di: Franco, Gabriel, et al.
Pubblicazione: (2024)

Finding Interpretable Prompt-Specific Circuits in Language Models
di: Franco, Gabriel, et al.
Pubblicazione: (2026)

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
di: Meng, Fanxu, et al.
Pubblicazione: (2024)

VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models
di: Hegde, Suhas G, et al.
Pubblicazione: (2025)

There is a Singularity in the Loss Landscape
di: Lowell, Mark
Pubblicazione: (2022)

Why Deep Jacobian Spectra Separate: Depth-Induced Scaling and Singular-Vector Alignment
di: Haas, Nathanaël, et al.
Pubblicazione: (2026)

Geometric Analysis of Token Selection in Multi-Head Attention
di: Mudarisov, Timur, et al.
Pubblicazione: (2026)

Do Attention Heads Compete or Cooperate during Counting?
di: Zsámboki, Pál, et al.
Pubblicazione: (2025)

Superiority of Multi-Head Attention in In-Context Linear Regression
di: Cui, Yingqian, et al.
Pubblicazione: (2024)

SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
di: Lingam, Vijay, et al.
Pubblicazione: (2024)

Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain
di: Wi, Hyowon, et al.
Pubblicazione: (2025)

Early Prediction of Sepsis: Feature-Aligned Transfer Learning
di: Komolafe, Oyindolapo O., et al.
Pubblicazione: (2025)

Quantifying LLM Attention-Head Stability: Implications for Circuit Universality
di: Bali, Karan, et al.
Pubblicazione: (2026)

The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms
di: Otsuka, Hikari, et al.
Pubblicazione: (2025)

Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
di: Ahmad, Areeb, et al.
Pubblicazione: (2025)

Attention as Binding: A Vector-Symbolic Perspective on Transformer Reasoning
di: Dhayalkar, Sahil Rajesh
Pubblicazione: (2025)

TransMLA: Multi-Head Latent Attention Is All You Need
di: Meng, Fanxu, et al.
Pubblicazione: (2025)

Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient
di: Wang, George, et al.
Pubblicazione: (2024)

MoH: Multi-Head Attention as Mixture-of-Head Attention
di: Jin, Peng, et al.
Pubblicazione: (2024)

Scaling Attention via Feature Sparsity
di: Xie, Yan, et al.
Pubblicazione: (2026)

Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
di: Li, Cheng, et al.
Pubblicazione: (2025)

Linear Transformers as VAR Models: Aligning Autoregressive Attention Mechanisms with Autoregressive Forecasting
di: Lu, Jiecheng, et al.
Pubblicazione: (2025)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
di: Yuan, Jingyang, et al.
Pubblicazione: (2025)

VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation
di: Sun, Yupeng, et al.
Pubblicazione: (2026)

Which Attention Heads Matter for In-Context Learning?
di: Yin, Kayo, et al.
Pubblicazione: (2025)

CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention
di: Zhou, Zhongzhu, et al.
Pubblicazione: (2026)

Even Heads Fix Odd Errors: Mechanistic Discovery and Surgical Repair in Transformer Attention
di: Sandoval, Gustavo
Pubblicazione: (2025)

Vectorized Attention with Learnable Encoding for Quantum Transformer
di: Guo, Ziqing, et al.
Pubblicazione: (2025)

Indirect Attention: Turning Context Misalignment into a Feature
di: Bahaduri, Bissmella, et al.
Pubblicazione: (2025)

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
di: Urrutia, Felipe, et al.
Pubblicazione: (2026)

How Many Heads Make an SSM? A Unified Framework for Attention and State Space Models
di: Ghodsi, Ali
Pubblicazione: (2025)

CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detection
di: Ortiz-Perez, David, et al.
Pubblicazione: (2025)

DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
di: Chen, Yilong, et al.
Pubblicazione: (2024)

The Anxiety of Influence: Bloom Filters in Transformer Attention Heads
di: Balogh, Peter
Pubblicazione: (2026)

Aligning Human and Machine Attention for Enhanced Supervised Learning
di: Chriqui, Avihay, et al.
Pubblicazione: (2025)

RESOLVE: Relational Reasoning with Symbolic and Object-Level Features Using Vector Symbolic Processing
di: Mejri, Mohamed, et al.
Pubblicazione: (2024)

Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
di: Xu, Yongzhong
Pubblicazione: (2026)

RecurFormer: Not All Transformer Heads Need Self-Attention
di: Yan, Ruiqing, et al.
Pubblicazione: (2024)

LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models
di: Jiang, Xinting, et al.
Pubblicazione: (2026)

Improving Steering Vectors by Targeting Sparse Autoencoder Features
di: Chalnev, Sviatoslav, et al.
Pubblicazione: (2024)