Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.15776 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916854120841216 |
|---|---|
| author | Sajid, Noor Medrano, Johan |
| author_facet | Sajid, Noor Medrano, Johan |
| contents | Parr et al., 2025 examines how auto-regressive and deep temporal models differ in their treatment of non-Markovian sequence modelling. Building on this, we highlight the need for dissociating model architectures, i.e., how the predictive distribution factorises, from the computations invoked at inference. We demonstrate that deep temporal computations are mimicked by autoregressive models by structuring context access during iterative inference. Using a transformer trained on next-token prediction, we show that inducing hierarchical temporal factorisation during iterative inference maintains predictive capacity while instantiating fewer computations. This emphasises that processes for constructing and refining predictions are not necessarily bound to their underlying model architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15776 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dissociating model architectures from inference computations Sajid, Noor Medrano, Johan Neurons and Cognition Computation and Language Machine Learning Parr et al., 2025 examines how auto-regressive and deep temporal models differ in their treatment of non-Markovian sequence modelling. Building on this, we highlight the need for dissociating model architectures, i.e., how the predictive distribution factorises, from the computations invoked at inference. We demonstrate that deep temporal computations are mimicked by autoregressive models by structuring context access during iterative inference. Using a transformer trained on next-token prediction, we show that inducing hierarchical temporal factorisation during iterative inference maintains predictive capacity while instantiating fewer computations. This emphasises that processes for constructing and refining predictions are not necessarily bound to their underlying model architectures. |
| title | Dissociating model architectures from inference computations |
| topic | Neurons and Cognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2507.15776 |