Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.23629 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914535952089088 |
|---|---|
| author | Hu, Sihan Cai, Xiansheng Huang, Yuan Yao, Zhiyuan Zhang, Linfeng Zhang, Pan Deng, Youjin Chen, Kun |
| author_facet | Hu, Sihan Cai, Xiansheng Huang, Yuan Yao, Zhiyuan Zhang, Linfeng Zhang, Pan Deng, Youjin Chen, Kun |
| contents | Reinforcement learning with verifiable rewards (RLVR) enables large language models to acquire slow, multi-step reasoning from sparse final-answer signals. We provide a statistical-physics picture of this emergence. We show that an autoregressive model's finite capacity forces it to compress its exponentially large prefix space into a Markov network of predictive states, on which slow thinking unfolds as a random walk -- the Concept Network (CoNet) picture. Within CoNet, RLVR dynamics are governed by two mechanisms: merging of compatible paths and frustrated competition among incompatible ones. Together they drive the network through nucleation, growth, and freezing into multi-input, single-output directed inverse trees. The picture reproduces the training dynamics of a 1.5-billion-parameter LLM and yields three predictions: reasoning chains lengthen as a geometric necessity of sparse topology; SFT induces catastrophic forgetting through bridge-node rupture; and frustration drives policy collapse. Building on the structural timing inherent in inverse-tree freezing, we propose Annealed-RLVR -- a brief SFT intervention at the moment of maximum frustration. It outperforms standard RLVR on both in- and out-of-distribution benchmarks, with the largest gains at high sampling budgets where standard RLVR collapses. The same SFT applied after the trees freeze instead triggers catastrophic forgetting, isolating timing as the active ingredient. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23629 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Emergent Slow Thinking in LLMs as Inverse Tree Freezing Hu, Sihan Cai, Xiansheng Huang, Yuan Yao, Zhiyuan Zhang, Linfeng Zhang, Pan Deng, Youjin Chen, Kun Artificial Intelligence Disordered Systems and Neural Networks Statistical Mechanics Machine Learning Physics and Society Reinforcement learning with verifiable rewards (RLVR) enables large language models to acquire slow, multi-step reasoning from sparse final-answer signals. We provide a statistical-physics picture of this emergence. We show that an autoregressive model's finite capacity forces it to compress its exponentially large prefix space into a Markov network of predictive states, on which slow thinking unfolds as a random walk -- the Concept Network (CoNet) picture. Within CoNet, RLVR dynamics are governed by two mechanisms: merging of compatible paths and frustrated competition among incompatible ones. Together they drive the network through nucleation, growth, and freezing into multi-input, single-output directed inverse trees. The picture reproduces the training dynamics of a 1.5-billion-parameter LLM and yields three predictions: reasoning chains lengthen as a geometric necessity of sparse topology; SFT induces catastrophic forgetting through bridge-node rupture; and frustration drives policy collapse. Building on the structural timing inherent in inverse-tree freezing, we propose Annealed-RLVR -- a brief SFT intervention at the moment of maximum frustration. It outperforms standard RLVR on both in- and out-of-distribution benchmarks, with the largest gains at high sampling budgets where standard RLVR collapses. The same SFT applied after the trees freeze instead triggers catastrophic forgetting, isolating timing as the active ingredient. |
| title | Emergent Slow Thinking in LLMs as Inverse Tree Freezing |
| topic | Artificial Intelligence Disordered Systems and Neural Networks Statistical Mechanics Machine Learning Physics and Society |
| url | https://arxiv.org/abs/2509.23629 |