Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914365231333376 |
|---|---|
| author | Yoshida, Haruto Kudo, Keito Aoki, Yoichi Tanaka, Ryota Saito, Itsumi Sakaguchi, Keisuke Inui, Kentaro |
| author_facet | Yoshida, Haruto Kudo, Keito Aoki, Yoichi Tanaka, Ryota Saito, Itsumi Sakaguchi, Keisuke Inui, Kentaro |
| contents | Large vision-language models (LVLMs) demonstrate strong performance on diagram understanding benchmarks, yet they still struggle with understanding relationships between elements, particularly those represented by nodes and directed edges (e.g., arrows and lines). To investigate the underlying causes of this limitation, we probe the internal representation of LVLMs using a carefully constructed synthetic diagram dataset based on directed graphs. Our probing experiments reveal that edge information is not linearly separable in the vision encoder and becomes linearly encoded only in the text tokens in the language model. In contrast, node information and global structural features are already linearly encoded in individual hidden states of the vision encoder. These findings suggest that the stage at which linearly separable representations are formed varies depending on the type of visual information. In particular, the delayed emergence of edge representations may help explain why LVLMs struggle with relational understanding, such as interpreting edge directions, which require more abstract, compositionally integrated processes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02865 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models Yoshida, Haruto Kudo, Keito Aoki, Yoichi Tanaka, Ryota Saito, Itsumi Sakaguchi, Keisuke Inui, Kentaro Computation and Language Computer Vision and Pattern Recognition Large vision-language models (LVLMs) demonstrate strong performance on diagram understanding benchmarks, yet they still struggle with understanding relationships between elements, particularly those represented by nodes and directed edges (e.g., arrows and lines). To investigate the underlying causes of this limitation, we probe the internal representation of LVLMs using a carefully constructed synthetic diagram dataset based on directed graphs. Our probing experiments reveal that edge information is not linearly separable in the vision encoder and becomes linearly encoded only in the text tokens in the language model. In contrast, node information and global structural features are already linearly encoded in individual hidden states of the vision encoder. These findings suggest that the stage at which linearly separable representations are formed varies depending on the type of visual information. In particular, the delayed emergence of edge representations may help explain why LVLMs struggle with relational understanding, such as interpreting edge directions, which require more abstract, compositionally integrated processes. |
| title | Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.02865 |