Cover Image

Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	O'Neill, Charles
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models
by: O'Neill, Charles, et al.
Published: (2024)

Type 2 Tobit Sample Selection Models with Bayesian Additive Regression Trees
by: O'Neill, Eoghan
Published: (2025)

Low-Rank Key Value Attention
by: O'Neill, James, et al.
Published: (2026)

Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse Autoencoders
by: O'Neill, Charles, et al.
Published: (2025)

Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity
by: Miller, Jack, et al.
Published: (2023)

Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders
by: O'Neill, Charles, et al.
Published: (2024)

Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
by: Mahadevan, Sridhar
Published: (2026)

Disentangling Dense Embeddings with Sparse Autoencoders
by: O'Neill, Charles, et al.
Published: (2024)

Sketching the Heat Kernel: Using Gaussian Processes to Embed Data
by: Gilbert, Anna C., et al.
Published: (2024)

From superposition to sparse codes: interpretable representations in neural networks
by: Klindt, David, et al.
Published: (2025)

Measuring Sharpness in Grokking
by: Miller, Jack, et al.
Published: (2024)

Self-Attention as Distributional Projection: A Unified Interpretation of Transformer Architecture
by: Mehta, Nihal
Published: (2025)

A Single Direction of Truth: An Observer Model's Linear Residual Probe Exposes and Steers Contextual Hallucinations
by: O'Neill, Charles, et al.
Published: (2025)

Modelling the Doughnut of social and planetary boundaries with frugal machine learning
by: Vrizzi, Stefano, et al.
Published: (2025)

CA-PCA: Manifold Dimension Estimation, Adapted for Curvature
by: Gilbert, Anna C., et al.
Published: (2023)

On the Universality of Transformer Architectures; How Much Attention Is Enough?
by: Abbasi, Amirreza, et al.
Published: (2025)

On Understanding Attention-Based In-Context Learning for Categorical Data
by: Wang, Aaron T., et al.
Published: (2024)

Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention
by: Evans, Ethan N., et al.
Published: (2024)

AnchorGT: Efficient and Flexible Attention Architecture for Scalable Graph Transformers
by: Zhu, Wenhao, et al.
Published: (2024)

Understanding Differential Transformer Unchains Pretrained Self-Attentions
by: Kong, Chaerin, et al.
Published: (2025)

Evaluating Sources: Strategies for Faculty-Librarian-Student Collaboration.
by: Simmons-O'Neill, Elizabeth
Published: (1990)

Graph Convolutions Enrich the Self-Attention in Transformers!
by: Choi, Jeongwhan, et al.
Published: (2023)

Quantifying Ambiguity in Categorical Annotations: A Measure and Statistical Inference Framework
by: Klugmann, Christopher, et al.
Published: (2025)

Geometry-Preserving Encoder/Decoder in Latent Generative Models
by: Lee, Wonjun, et al.
Published: (2025)

Local Attention Mechanism: Boosting the Transformer Architecture for Long-Sequence Time Series Forecasting
by: Aguilera-Martos, Ignacio, et al.
Published: (2024)

Parametrized Multi-Agent Routing via Deep Attention Models
by: Basiri, Salar, et al.
Published: (2025)

Self-Supervised Transformer Architecture for Change Detection in Radio Access Networks
by: Kozlov, Igor, et al.
Published: (2023)

DEFault++: Automated Fault Detection, Categorization, and Diagnosis for Transformer Architectures
by: Jahan, Sigma, et al.
Published: (2026)

CAST: Clustering Self-Attention using Surrogate Tokens for Efficient Transformers
by: van Engelenhoven, Adjorn, et al.
Published: (2024)

Triple Attention Transformer Architecture for Time-Dependent Concrete Creep Prediction
by: Dokduea, Warayut, et al.
Published: (2025)

Multistability of Self-Attention Dynamics in Transformers
by: Altafini, Claudio
Published: (2025)

Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliability
by: Adhikari, Santosh Premi, et al.
Published: (2026)

Cutting Through the Noise: On-the-fly Outlier Detection for Robust Training of Machine Learning Interatomic Potentials
by: Lam, Terry C. W., et al.
Published: (2026)

Transformers for Tabular Data: A Training Perspective of Self-Attention via Optimal Transport
by: Quadrio, Alessandro, et al.
Published: (2025)

Quantum Adaptive Self-Attention for Quantum Transformer Models
by: Chen, Chi-Sheng, et al.
Published: (2025)

A Markov Categorical Framework for Language Modeling
by: Zhang, Yifan
Published: (2025)

Spacetime $E(n)$-Transformer: Equivariant Attention for Spatio-temporal Graphs
by: Charles, Sergio G.
Published: (2024)

EEG Emotion Classification Using an Enhanced Transformer-CNN-BiLSTM Architecture with Dual Attention Mechanisms
by: Karim, S M Rakib UI, et al.
Published: (2026)

Faster Transformer Decoding: N-gram Masked Self-Attention
by: Chelba, Ciprian, et al.
Published: (2020)

Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention
by: Guo, Zhenyu, et al.
Published: (2025)