Deriving Transformer Architectures as Implicit Multinomial Regression
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908613911511040 |
|---|---|
| author | Actor, Jonas A. Gruber, Anthony Cyr, Eric C. |
| author_facet | Actor, Jonas A. Gruber, Anthony Cyr, Eric C. |
| contents | While attention has been empirically shown to improve model performance, it lacks a rigorous mathematical justification. This short paper establishes a novel connection between attention mechanisms and multinomial regression. Specifically, we show that in a fixed multinomial regression setting, optimizing over latent features yields solutions that align with the dynamics induced on features by attention blocks. In other words, the evolution of representations through a transformer can be interpreted as a trajectory that recovers the optimal features for classification. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_04653 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Deriving Transformer Architectures as Implicit Multinomial Regression Actor, Jonas A. Gruber, Anthony Cyr, Eric C. Machine Learning Artificial Intelligence Numerical Analysis While attention has been empirically shown to improve model performance, it lacks a rigorous mathematical justification. This short paper establishes a novel connection between attention mechanisms and multinomial regression. Specifically, we show that in a fixed multinomial regression setting, optimizing over latent features yields solutions that align with the dynamics induced on features by attention blocks. In other words, the evolution of representations through a transformer can be interpreted as a trajectory that recovers the optimal features for classification. |
| title | Deriving Transformer Architectures as Implicit Multinomial Regression |
| topic | Machine Learning Artificial Intelligence Numerical Analysis |
| url | https://arxiv.org/abs/2509.04653 |