Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909661927571456 |
|---|---|
| author | Someya, Taiga Yoshida, Ryo Yanaka, Hitomi Oseki, Yohei |
| author_facet | Someya, Taiga Yoshida, Ryo Yanaka, Hitomi Oseki, Yohei |
| contents | Recent work has demonstrated that neural language models encode syntactic structures in their internal representations, yet the derivations by which these structures are constructed across layers remain poorly understood. In this paper, we propose Derivational Probing to investigate how micro-syntactic structures (e.g., subject noun phrases) and macro-syntactic structures (e.g., the relationship between the root verbs and their direct dependents) are constructed as word embeddings propagate upward across layers. Our experiments on BERT reveal a clear bottom-up derivation: micro-syntactic structures emerge in lower layers and are gradually integrated into a coherent macro-syntactic structure in higher layers. Furthermore, a targeted evaluation on subject-verb number agreement shows that the timing of constructing macro-syntactic structures is critical for downstream performance, suggesting an optimal timing for integrating global syntactic information. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_21861 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models Someya, Taiga Yoshida, Ryo Yanaka, Hitomi Oseki, Yohei Computation and Language Recent work has demonstrated that neural language models encode syntactic structures in their internal representations, yet the derivations by which these structures are constructed across layers remain poorly understood. In this paper, we propose Derivational Probing to investigate how micro-syntactic structures (e.g., subject noun phrases) and macro-syntactic structures (e.g., the relationship between the root verbs and their direct dependents) are constructed as word embeddings propagate upward across layers. Our experiments on BERT reveal a clear bottom-up derivation: micro-syntactic structures emerge in lower layers and are gradually integrated into a coherent macro-syntactic structure in higher layers. Furthermore, a targeted evaluation on subject-verb number agreement shows that the timing of constructing macro-syntactic structures is critical for downstream performance, suggesting an optimal timing for integrating global syntactic information. |
| title | Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.21861 |