Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.23314 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918200001691648 |
|---|---|
| author | Pappone, Francesco Crisostomi, Donato Rodolà, Emanuele |
| author_facet | Pappone, Francesco Crisostomi, Donato Rodolà, Emanuele |
| contents | Recurrent-depth transformers scale test-time compute by iterating latent computations before emitting tokens. We study the geometry of these iterates and argue for a simple, two-scale operational picture: (i) within a looped block, updates act as small-scale refinements; (ii) across consecutive blocks, states undergo a larger-scale drift. Across training, our measurements show that loop steps become smaller and increasingly orthogonal to one another, indicating better local modeling of fine structure rather than merely pushing in a single direction. These dynamics motivate an early-exit mechanism based on the model's second-order difference in step-size, which we show is superior in terms of performance, stability and time-efficiency, when compared to the KL-divergence exit strategy of Geiping et al. and its naive first-order counterpart. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23314 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Two-Scale Latent Dynamics for Recurrent-Depth Transformers Pappone, Francesco Crisostomi, Donato Rodolà, Emanuele Machine Learning Recurrent-depth transformers scale test-time compute by iterating latent computations before emitting tokens. We study the geometry of these iterates and argue for a simple, two-scale operational picture: (i) within a looped block, updates act as small-scale refinements; (ii) across consecutive blocks, states undergo a larger-scale drift. Across training, our measurements show that loop steps become smaller and increasingly orthogonal to one another, indicating better local modeling of fine structure rather than merely pushing in a single direction. These dynamics motivate an early-exit mechanism based on the model's second-order difference in step-size, which we show is superior in terms of performance, stability and time-efficiency, when compared to the KL-divergence exit strategy of Geiping et al. and its naive first-order counterpart. |
| title | Two-Scale Latent Dynamics for Recurrent-Depth Transformers |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.23314 |