Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910832566206464 |
|---|---|
| author | Geiping, Jonas McLeish, Sean Jain, Neel Kirchenbauer, John Singh, Siddharth Bartoldson, Brian R. Kailkhura, Bhavya Bhatele, Abhinav Goldstein, Tom |
| author_facet | Geiping, Jonas McLeish, Sean Jain, Neel Kirchenbauer, John Singh, Siddharth Bartoldson, Brian R. Kailkhura, Bhavya Bhatele, Abhinav Goldstein, Tom |
| contents | We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. We scale a proof-of-concept model to 3.5 billion parameters and 800 billion tokens. We show that the resulting model can improve its performance on reasoning benchmarks, sometimes dramatically, up to a computation load equivalent to 50 billion parameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_05171 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach Geiping, Jonas McLeish, Sean Jain, Neel Kirchenbauer, John Singh, Siddharth Bartoldson, Brian R. Kailkhura, Bhavya Bhatele, Abhinav Goldstein, Tom Machine Learning Computation and Language We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. We scale a proof-of-concept model to 3.5 billion parameters and 800 billion tokens. We show that the resulting model can improve its performance on reasoning benchmarks, sometimes dramatically, up to a computation load equivalent to 50 billion parameters. |
| title | Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.05171 |