StructLens: A Structural Lens for Language Models via Maximum Spanning Trees
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917507293511680 |
|---|---|
| author | Sakajo, Haruki Hudi, Frederikus Sakai, Yusuke Kamigaito, Hidetaka Watanabe, Taro |
| author_facet | Sakajo, Haruki Hudi, Frederikus Sakai, Yusuke Kamigaito, Hidetaka Watanabe, Taro |
| contents | Language exhibits inherent structures, a property that explains both language acquisition and language change. Given this characteristic, we expect language models to manifest their own internal structures as well. While interpretability research has investigated how models compute representations mechanistically through attention patterns and Sparse AutoEncoders, the organization of the resulting representations is overlooked. To address this gap, we introduce StructLens, a framework to analyze representations through a holistic structural view. StructLens constructs maximum spanning trees based on the semantic representations in residual streams, inspired by tree representation in dependency parsing, and provides summaries of token relationships in representation space. We analyze how contiguous tokens are also nearby in representation space and find that middle layers show the strongest local-span organization. Moreover, analysis of pre-training checkpoints reveals that smaller local units become detectable earlier in pre-training, and larger units later. Our findings demonstrate that StructLens provides insights into how models organize token representations across layers and training. Our code is available at https://github.com/naist-nlp/structlens. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_03328 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | StructLens: A Structural Lens for Language Models via Maximum Spanning Trees Sakajo, Haruki Hudi, Frederikus Sakai, Yusuke Kamigaito, Hidetaka Watanabe, Taro Computation and Language Artificial Intelligence Language exhibits inherent structures, a property that explains both language acquisition and language change. Given this characteristic, we expect language models to manifest their own internal structures as well. While interpretability research has investigated how models compute representations mechanistically through attention patterns and Sparse AutoEncoders, the organization of the resulting representations is overlooked. To address this gap, we introduce StructLens, a framework to analyze representations through a holistic structural view. StructLens constructs maximum spanning trees based on the semantic representations in residual streams, inspired by tree representation in dependency parsing, and provides summaries of token relationships in representation space. We analyze how contiguous tokens are also nearby in representation space and find that middle layers show the strongest local-span organization. Moreover, analysis of pre-training checkpoints reveals that smaller local units become detectable earlier in pre-training, and larger units later. Our findings demonstrate that StructLens provides insights into how models organize token representations across layers and training. Our code is available at https://github.com/naist-nlp/structlens. |
| title | StructLens: A Structural Lens for Language Models via Maximum Spanning Trees |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.03328 |