What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911127058776064 |
|---|---|
| author | Chang, Heng-Sheng Mehta, Prashant G. |
| author_facet | Chang, Heng-Sheng Mehta, Prashant G. |
| contents | In the 1940s, Wiener introduced a linear predictor, where the future prediction is computed by linearly combining the past data. A transformer generalizes this idea: it is a nonlinear predictor where the next-token prediction is computed by nonlinearly combining the past tokens. In this essay, we present a probabilistic model that interprets transformer signals as surrogates of conditional measures, and layer operations as fixed-point updates. An explicit form of the fixed-point update is described for the special case when the probabilistic model is a hidden Markov model (HMM). In part, this paper is in an attempt to bridge the classical nonlinear filtering theory with modern inference architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_20211 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture Chang, Heng-Sheng Mehta, Prashant G. Machine Learning Systems and Control Probability In the 1940s, Wiener introduced a linear predictor, where the future prediction is computed by linearly combining the past data. A transformer generalizes this idea: it is a nonlinear predictor where the next-token prediction is computed by nonlinearly combining the past tokens. In this essay, we present a probabilistic model that interprets transformer signals as surrogates of conditional measures, and layer operations as fixed-point updates. An explicit form of the fixed-point update is described for the special case when the probabilistic model is a hidden Markov model (HMM). In part, this paper is in an attempt to bridge the classical nonlinear filtering theory with modern inference architectures. |
| title | What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture |
| topic | Machine Learning Systems and Control Probability |
| url | https://arxiv.org/abs/2508.20211 |