Cover Image

Self-Attention as Distributional Projection: A Unified Interpretation of Transformer Architecture

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	Mehta, Nihal
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures
by: O'Neill, Charles
Published: (2025)

When Are Learning Biases Equivalent? A Unifying Framework for Fairness, Robustness, and Distribution Shift
by: Mehta, Sushant
Published: (2025)

Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
by: Su, Zunhai, et al.
Published: (2026)

Adaptive Two Sided Laplace Transforms: A Learnable, Interpretable, and Scalable Replacement for Self-Attention
by: Kiruluta, Andrew
Published: (2025)

CardioPatternFormer: Pattern-Guided Attention for Interpretable ECG Classification with Transformer Architecture
by: Uğraş, Berat Kutay, et al.
Published: (2025)

Projection-Free Transformers via Gaussian Kernel Attention
by: Kundu, Debarshi, et al.
Published: (2026)

On the Universality of Transformer Architectures; How Much Attention Is Enough?
by: Abbasi, Amirreza, et al.
Published: (2025)

Dual Filter: A Transformer-like Inference Architecture for Hidden Markov Models
by: Chang, Heng-Sheng, et al.
Published: (2025)

Interpretable-by-Design Transformers via Architectural Stream Independence
by: Kerce, Clayton, et al.
Published: (2026)

Self-Ablating Transformers: More Interpretability, Less Sparsity
by: Ferrao, Jeremias, et al.
Published: (2025)

Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers
by: Aggarwal, Shubham, et al.
Published: (2026)

Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention
by: Evans, Ethan N., et al.
Published: (2024)

AnchorGT: Efficient and Flexible Attention Architecture for Scalable Graph Transformers
by: Zhu, Wenhao, et al.
Published: (2024)

Understanding Differential Transformer Unchains Pretrained Self-Attentions
by: Kong, Chaerin, et al.
Published: (2025)

TrialDura: Hierarchical Attention Transformer for Interpretable Clinical Trial Duration Prediction
by: Yue, Ling, et al.
Published: (2024)

Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition
by: Xu, Haoren, et al.
Published: (2026)

Towards Mechanistic Interpretability of Graph Transformers via Attention Graphs
by: El, Batu, et al.
Published: (2025)

Scaling Laws and In-Context Learning: A Unified Theoretical Framework
by: Mehta, Sushant, et al.
Published: (2025)

Unveiling and Controlling Anomalous Attention Distribution in Transformers
by: Yan, Ruiqing, et al.
Published: (2024)

Graph Convolutions Enrich the Self-Attention in Transformers!
by: Choi, Jeongwhan, et al.
Published: (2023)

FlashOmni: A Unified Sparse Attention Engine for Diffusion Transformers
by: Qiao, Liang, et al.
Published: (2025)

Local Attention Mechanism: Boosting the Transformer Architecture for Long-Sequence Time Series Forecasting
by: Aguilera-Martos, Ignacio, et al.
Published: (2024)

DARTS-GT: Differentiable Architecture Search for Graph Transformers with Quantifiable Instance-Specific Interpretability Analysis
by: Chakraborty, Shruti Sarika, et al.
Published: (2025)

Self-Supervised Transformer Architecture for Change Detection in Radio Access Networks
by: Kozlov, Igor, et al.
Published: (2023)

Triple Attention Transformer Architecture for Time-Dependent Concrete Creep Prediction
by: Dokduea, Warayut, et al.
Published: (2025)

CAST: Clustering Self-Attention using Surrogate Tokens for Efficient Transformers
by: van Engelenhoven, Adjorn, et al.
Published: (2024)

Multistability of Self-Attention Dynamics in Transformers
by: Altafini, Claudio
Published: (2025)

Transformers for Tabular Data: A Training Perspective of Self-Attention via Optimal Transport
by: Quadrio, Alessandro, et al.
Published: (2025)

The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling
by: Kerce, J. Clayton, et al.
Published: (2026)

Quantum Adaptive Self-Attention for Quantum Transformer Models
by: Chen, Chi-Sheng, et al.
Published: (2025)

ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
by: Helbling, Alec, et al.
Published: (2025)

Chessformer: A Unified Architecture for Chess Modeling
by: Monroe, Daniel, et al.
Published: (2026)

Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights
by: Ho, Sy-Tuyen, et al.
Published: (2025)

Small Language Models for Agentic Systems: A Survey of Architectures, Capabilities, and Deployment Trade offs
by: Sharma, Raghav, et al.
Published: (2025)

CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting
by: Pati, Viresh, et al.
Published: (2026)

Probing Information Distribution in Transformer Architectures through Entropy Analysis
by: Buonanno, Amedeo, et al.
Published: (2025)

Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
by: Mahadevan, Sridhar
Published: (2026)

EEG Emotion Classification Using an Enhanced Transformer-CNN-BiLSTM Architecture with Dual Attention Mechanisms
by: Karim, S M Rakib UI, et al.
Published: (2026)

Interpretable Tensor Fusion
by: Varshneya, Saurabh, et al.
Published: (2024)

Faster Transformer Decoding: N-gram Masked Self-Attention
by: Chelba, Ciprian, et al.
Published: (2020)