SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917421996048384 |
|---|---|
| author | Yu, Chengting Shu, Xiaobo Wang, Yadao Zhang, Yizhen Wu, Haoyi Wu, You Long, Rujiao Chen, Ziheng Xu, Yuchi Su, Wenbo Zheng, Bo |
| author_facet | Yu, Chengting Shu, Xiaobo Wang, Yadao Zhang, Yizhen Wu, Haoyi Wu, You Long, Rujiao Chen, Ziheng Xu, Yuchi Su, Wenbo Zheng, Bo |
| contents | Recursive (looped) Transformers decouple computational depth from parameter depth by repeatedly applying shared layers, providing an explicit architectural primitive for iterative refinement and latent reasoning. However, early looped Transformers often underperform non-recursive baselines of equal compute. While recent literature has introduced more effective recursion mechanisms to mitigate this gap, existing architectures still operate at a fixed, full-token resolution, neglecting the potential efficiency of computing over compressed latent representations. In this paper, we propose SpiralFormer, a looped Transformer that executes recurrence under a multi-resolution recursion schedule. We provide probing evidence that multi-resolution recursion enables the model to learn hierarchical dependencies by inducing iteration-wise functional specialization across different scales. Empirically, SpiralFormer achieves better parameter and compute efficiency than both looped and non-looped baselines across model scales from 160M to 1.4B, establishing sequence resolution as a potential axis for scaling recursive architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_11698 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion Yu, Chengting Shu, Xiaobo Wang, Yadao Zhang, Yizhen Wu, Haoyi Wu, You Long, Rujiao Chen, Ziheng Xu, Yuchi Su, Wenbo Zheng, Bo Machine Learning Recursive (looped) Transformers decouple computational depth from parameter depth by repeatedly applying shared layers, providing an explicit architectural primitive for iterative refinement and latent reasoning. However, early looped Transformers often underperform non-recursive baselines of equal compute. While recent literature has introduced more effective recursion mechanisms to mitigate this gap, existing architectures still operate at a fixed, full-token resolution, neglecting the potential efficiency of computing over compressed latent representations. In this paper, we propose SpiralFormer, a looped Transformer that executes recurrence under a multi-resolution recursion schedule. We provide probing evidence that multi-resolution recursion enables the model to learn hierarchical dependencies by inducing iteration-wise functional specialization across different scales. Empirically, SpiralFormer achieves better parameter and compute efficiency than both looped and non-looped baselines across model scales from 160M to 1.4B, establishing sequence resolution as a potential axis for scaling recursive architectures. |
| title | SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.11698 |