Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.17869 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910000726671360 |
|---|---|
| author | Chen, Michelle Chao Miller, Moritz Schölkopf, Bernhard Guo, Siyuan |
| author_facet | Chen, Michelle Chao Miller, Moritz Schölkopf, Bernhard Guo, Siyuan |
| contents | Learning structural information from observational data is central to producing new knowledge outside the training corpus. This holds for mechanistic understanding in scientific discovery as well as flexible test-time compositional generation. We thus study how language models learn abstract structures and utilize the learnt structural information at test-time. To ensure a controlled setup, we design a natural language dataset based on linguistic structural transformations. We empirically show that the emergence of learning structural information correlates with complex reasoning tasks, and that the ability to perform test-time compositional generation remains limited. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_17869 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | On the Emergence and Test-Time Use of Structural Information in Large Language Models Chen, Michelle Chao Miller, Moritz Schölkopf, Bernhard Guo, Siyuan Computation and Language Machine Learning Learning structural information from observational data is central to producing new knowledge outside the training corpus. This holds for mechanistic understanding in scientific discovery as well as flexible test-time compositional generation. We thus study how language models learn abstract structures and utilize the learnt structural information at test-time. To ensure a controlled setup, we design a natural language dataset based on linguistic structural transformations. We empirically show that the emergence of learning structural information correlates with complex reasoning tasks, and that the ability to perform test-time compositional generation remains limited. |
| title | On the Emergence and Test-Time Use of Structural Information in Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2601.17869 |