Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917205505998848 |
|---|---|
| author | Yamagiwa, Hiroaki Takase, Yusuke Shimodaira, Hidetoshi |
| author_facet | Yamagiwa, Hiroaki Takase, Yusuke Shimodaira, Hidetoshi |
| contents | Understanding relationships between attention heads is essential for interpreting the internal structure of Transformers, yet existing metrics do not capture this structure well. We focus on the subspaces spanned by attention-head weight matrices and quantify head-to-head relationships using the Projection Kernel (PK), a principal-angle-based measure of subspace similarity. Experiments show that PK reproduces known head-to-head interactions on the IOI task more clearly than prior metrics such as the Composition Score. We further introduce a framework to quantify the informativeness of PK distributions by comparing them with a reference distribution derived from random orthogonal subspaces. As an application, we analyze a directed graph constructed from PK and show that, in GPT2-small, L4H7 acts as a hub by functioning as an identity head. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_10266 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel Yamagiwa, Hiroaki Takase, Yusuke Shimodaira, Hidetoshi Computation and Language Understanding relationships between attention heads is essential for interpreting the internal structure of Transformers, yet existing metrics do not capture this structure well. We focus on the subspaces spanned by attention-head weight matrices and quantify head-to-head relationships using the Projection Kernel (PK), a principal-angle-based measure of subspace similarity. Experiments show that PK reproduces known head-to-head interactions on the IOI task more clearly than prior metrics such as the Composition Score. We further introduce a framework to quantify the informativeness of PK distributions by comparing them with a reference distribution derived from random orthogonal subspaces. As an application, we analyze a directed graph constructed from PK and show that, in GPT2-small, L4H7 acts as a hub by functioning as an identity head. |
| title | Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.10266 |