Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2509.04653 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908613911511040 |
|---|---|
| author | Actor, Jonas A. Gruber, Anthony Cyr, Eric C. |
| author_facet | Actor, Jonas A. Gruber, Anthony Cyr, Eric C. |
| contents | While attention has been empirically shown to improve model performance, it lacks a rigorous mathematical justification. This short paper establishes a novel connection between attention mechanisms and multinomial regression. Specifically, we show that in a fixed multinomial regression setting, optimizing over latent features yields solutions that align with the dynamics induced on features by attention blocks. In other words, the evolution of representations through a transformer can be interpreted as a trajectory that recovers the optimal features for classification. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_04653 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Deriving Transformer Architectures as Implicit Multinomial Regression Actor, Jonas A. Gruber, Anthony Cyr, Eric C. Machine Learning Artificial Intelligence Numerical Analysis While attention has been empirically shown to improve model performance, it lacks a rigorous mathematical justification. This short paper establishes a novel connection between attention mechanisms and multinomial regression. Specifically, we show that in a fixed multinomial regression setting, optimizing over latent features yields solutions that align with the dynamics induced on features by attention blocks. In other words, the evolution of representations through a transformer can be interpreted as a trajectory that recovers the optimal features for classification. |
| title | Deriving Transformer Architectures as Implicit Multinomial Regression |
| topic | Machine Learning Artificial Intelligence Numerical Analysis |
| url | https://arxiv.org/abs/2509.04653 |