Geometric Signatures of Compositionality Across a Language Model's Lifetime
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918060873482240 |
|---|---|
| author | Lee, Jin Hwa Jiralerspong, Thomas Yu, Lei Bengio, Yoshua Cheng, Emily |
| author_facet | Lee, Jin Hwa Jiralerspong, Thomas Yu, Lei Bengio, Yoshua Cheng, Emily |
| contents | By virtue of linguistic compositionality, few syntactic rules and a finite lexicon can generate an unbounded number of sentences. That is, language, though seemingly high-dimensional, can be explained using relatively few degrees of freedom. An open question is whether contemporary language models (LMs) reflect the intrinsic simplicity of language that is enabled by compositionality. We take a geometric view of this problem by relating the degree of compositionality in a dataset to the intrinsic dimension (ID) of its representations under an LM, a measure of feature complexity. We find not only that the degree of dataset compositionality is reflected in representations' ID, but that the relationship between compositionality and geometric complexity arises due to learned linguistic features over training. Finally, our analyses reveal a striking contrast between nonlinear and linear dimensionality, showing they respectively encode semantic and superficial aspects of linguistic composition. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01444 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Geometric Signatures of Compositionality Across a Language Model's Lifetime Lee, Jin Hwa Jiralerspong, Thomas Yu, Lei Bengio, Yoshua Cheng, Emily Computation and Language Artificial Intelligence Information Theory Machine Learning By virtue of linguistic compositionality, few syntactic rules and a finite lexicon can generate an unbounded number of sentences. That is, language, though seemingly high-dimensional, can be explained using relatively few degrees of freedom. An open question is whether contemporary language models (LMs) reflect the intrinsic simplicity of language that is enabled by compositionality. We take a geometric view of this problem by relating the degree of compositionality in a dataset to the intrinsic dimension (ID) of its representations under an LM, a measure of feature complexity. We find not only that the degree of dataset compositionality is reflected in representations' ID, but that the relationship between compositionality and geometric complexity arises due to learned linguistic features over training. Finally, our analyses reveal a striking contrast between nonlinear and linear dimensionality, showing they respectively encode semantic and superficial aspects of linguistic composition. |
| title | Geometric Signatures of Compositionality Across a Language Model's Lifetime |
| topic | Computation and Language Artificial Intelligence Information Theory Machine Learning |
| url | https://arxiv.org/abs/2410.01444 |