How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915985834901504 |
|---|---|
| author | Schwethelm, Kristian Rueckert, Daniel Kaissis, Georgios |
| author_facet | Schwethelm, Kristian Rueckert, Daniel Kaissis, Georgios |
| contents | We measure how much one recurrence is worth to a looped (depth-recurrent) transformer, in equivalent unique parameters. From an iso-depth pretraining sweep across recurrence counts $r \in \{1, 2, 4, 8\}$ spanning ${\sim}50\times$ in training compute, we fit a joint scaling law $L = E + A\,(N_\text{once} + r^φ N_\text{rec})^{-α} + B\,D^{-β}$ and measure a recurrence-equivalence exponent $φ= 0.46$. Intuitively, $φ$ tells us whether looping a block $r$ times is equivalent in validation loss to $r$ unique blocks of a non-looped model (full equivalence, $φ{=}1$) or to a single block run repeatedly with no capacity gain ($φ{=}0$). Our $φ= 0.46$ sits in between, so replacing unique blocks with shared recurrences increases validation loss at matched training compute. For example, at $r{=}4$ a 410M looped model performs on par with a 580M non-looped model, but incurs the training cost of a 1B non-looped one. We demonstrate the utility of $φ$ as a diagnostic tool on two case studies: commonly used truncated backpropagation lowers $φ$ to $0.38$, indicating that the loop mechanism is poorly trained under truncation, even though validation loss decreases. Conversely, hyperconnections raise $φ$ to $0.65$, a genuine capacity gain. Our method separates true loop improvements from training-side gains, a distinction raw validation loss cannot make. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_21106 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models Schwethelm, Kristian Rueckert, Daniel Kaissis, Georgios Machine Learning Computation and Language We measure how much one recurrence is worth to a looped (depth-recurrent) transformer, in equivalent unique parameters. From an iso-depth pretraining sweep across recurrence counts $r \in \{1, 2, 4, 8\}$ spanning ${\sim}50\times$ in training compute, we fit a joint scaling law $L = E + A\,(N_\text{once} + r^φ N_\text{rec})^{-α} + B\,D^{-β}$ and measure a recurrence-equivalence exponent $φ= 0.46$. Intuitively, $φ$ tells us whether looping a block $r$ times is equivalent in validation loss to $r$ unique blocks of a non-looped model (full equivalence, $φ{=}1$) or to a single block run repeatedly with no capacity gain ($φ{=}0$). Our $φ= 0.46$ sits in between, so replacing unique blocks with shared recurrences increases validation loss at matched training compute. For example, at $r{=}4$ a 410M looped model performs on par with a 580M non-looped model, but incurs the training cost of a 1B non-looped one. We demonstrate the utility of $φ$ as a diagnostic tool on two case studies: commonly used truncated backpropagation lowers $φ$ to $0.38$, indicating that the loop mechanism is poorly trained under truncation, even though validation loss decreases. Conversely, hyperconnections raise $φ$ to $0.65$, a genuine capacity gain. Our method separates true loop improvements from training-side gains, a distinction raw validation loss cannot make. |
| title | How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2604.21106 |