Reasoning Models Know What's Important, and Encode It in Their Activations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908980299694080 |
|---|---|
| author | Nikankin, Yaniv Tutek, Martin Ashuach, Tomer Rosenfeld, Jonathan Belinkov, Yonatan |
| author_facet | Nikankin, Yaniv Tutek, Martin Ashuach, Tomer Rosenfeld, Jonathan Belinkov, Yonatan |
| contents | Language models often solve complex tasks by generating long reasoning chains, consisting of many steps with varying importance. While some steps are crucial for generating the final answer, others are removable. Determining which steps matter most, and why, remains an open question central to understanding how models process reasoning. We investigate if this question is best approached through model internals or through tokens of the reasoning chain itself. We find that model activations contain more information than tokens for identifying important reasoning steps. Crucially, by training probes on model activations to predict importance, we show that models encode an internal representation of step importance, even prior to the generation of subsequent steps. This internal representation of importance generalizes across models, is distributed across layers, and does not correlate with surface-level features, such as a step's relative position or its length. Our findings suggest that analyzing activations can reveal aspects of reasoning that surface-level approaches fundamentally miss, indicating that reasoning analyses should look into model internals. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_18307 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Reasoning Models Know What's Important, and Encode It in Their Activations Nikankin, Yaniv Tutek, Martin Ashuach, Tomer Rosenfeld, Jonathan Belinkov, Yonatan Computation and Language 68T5 I.2.7 Language models often solve complex tasks by generating long reasoning chains, consisting of many steps with varying importance. While some steps are crucial for generating the final answer, others are removable. Determining which steps matter most, and why, remains an open question central to understanding how models process reasoning. We investigate if this question is best approached through model internals or through tokens of the reasoning chain itself. We find that model activations contain more information than tokens for identifying important reasoning steps. Crucially, by training probes on model activations to predict importance, we show that models encode an internal representation of step importance, even prior to the generation of subsequent steps. This internal representation of importance generalizes across models, is distributed across layers, and does not correlate with surface-level features, such as a step's relative position or its length. Our findings suggest that analyzing activations can reveal aspects of reasoning that surface-level approaches fundamentally miss, indicating that reasoning analyses should look into model internals. |
| title | Reasoning Models Know What's Important, and Encode It in Their Activations |
| topic | Computation and Language 68T5 I.2.7 |
| url | https://arxiv.org/abs/2604.18307 |