Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917071349088256 |
|---|---|
| author | McLeish, Sean Li, Ang Kirchenbauer, John Kalra, Dayal Singh Bartoldson, Brian R. Kailkhura, Bhavya Schwarzschild, Avi Geiping, Jonas Goldstein, Tom Goldblum, Micah |
| author_facet | McLeish, Sean Li, Ang Kirchenbauer, John Kalra, Dayal Singh Bartoldson, Brian R. Kailkhura, Bhavya Schwarzschild, Avi Geiping, Jonas Goldstein, Tom Goldblum, Micah |
| contents | Recent advances in depth-recurrent language models show that recurrence can decouple train-time compute and parameter count from test-time compute. In this work, we study how to convert existing pretrained non-recurrent language models into depth-recurrent models. We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost. In our experiments, on mathematics, we observe that converting pretrained models to recurrent ones results in better performance at a given compute budget than simply post-training the original non-recurrent language model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_07384 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence McLeish, Sean Li, Ang Kirchenbauer, John Kalra, Dayal Singh Bartoldson, Brian R. Kailkhura, Bhavya Schwarzschild, Avi Geiping, Jonas Goldstein, Tom Goldblum, Micah Computation and Language Artificial Intelligence Machine Learning Recent advances in depth-recurrent language models show that recurrence can decouple train-time compute and parameter count from test-time compute. In this work, we study how to convert existing pretrained non-recurrent language models into depth-recurrent models. We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost. In our experiments, on mathematics, we observe that converting pretrained models to recurrent ones results in better performance at a given compute budget than simply post-training the original non-recurrent language model. |
| title | Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2511.07384 |