Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2508.16745 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909020681404416 |
|---|---|
| author | Rodkin, Ivan Orel, Daniil Smirnov, Konstantin Bolatov, Arman Elbouardi, Bilal Hassan, Besher Kuratov, Yuri Bulatov, Aydar Nakov, Preslav Baldwin, Timothy Shelmanov, Artem Burtsev, Mikhail |
| author_facet | Rodkin, Ivan Orel, Daniil Smirnov, Konstantin Bolatov, Arman Elbouardi, Bilal Hassan, Besher Kuratov, Yuri Bulatov, Aydar Nakov, Preslav Baldwin, Timothy Shelmanov, Artem Burtsev, Mikhail |
| contents | Reasoning is a core capability of large language models, yet how multi-step reasoning is learned and executed remains unclear. We study this question in a controlled cellular-automata (1dCA) framework that excludes memorisation by using disjoint training and test rules. Given a short state sequence, the model is required to infer the hidden local rule and then chain it to predict multiple future steps. Our evaluation shows that LLMs largely fail to reliably solve a natural-language proxy of the proposed task. We find that most neural architectures trained from scratch can learn rule inference and achieve high next-step accuracy, but performance drops sharply as the required number of intermediate reasoning steps increases. Experiments show that increasing model depth is crucial, and extending effective depth via recurrence, memory, or test-time compute improves results but remains bounded. The code is available on github: https://github.com/RodkinIvan/associative-recurrent-memory-transformer/tree/ACT |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_16745 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling Rodkin, Ivan Orel, Daniil Smirnov, Konstantin Bolatov, Arman Elbouardi, Bilal Hassan, Besher Kuratov, Yuri Bulatov, Aydar Nakov, Preslav Baldwin, Timothy Shelmanov, Artem Burtsev, Mikhail Machine Learning Artificial Intelligence Reasoning is a core capability of large language models, yet how multi-step reasoning is learned and executed remains unclear. We study this question in a controlled cellular-automata (1dCA) framework that excludes memorisation by using disjoint training and test rules. Given a short state sequence, the model is required to infer the hidden local rule and then chain it to predict multiple future steps. Our evaluation shows that LLMs largely fail to reliably solve a natural-language proxy of the proposed task. We find that most neural architectures trained from scratch can learn rule inference and achieve high next-step accuracy, but performance drops sharply as the required number of intermediate reasoning steps increases. Experiments show that increasing model depth is crucial, and extending effective depth via recurrence, memory, or test-time compute improves results but remains bounded. The code is available on github: https://github.com/RodkinIvan/associative-recurrent-memory-transformer/tree/ACT |
| title | Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.16745 |