Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918152028291072 |
|---|---|
| author | Scivetti, Wesley Aoyama, Tatsuya Wilcox, Ethan Schneider, Nathan |
| author_facet | Scivetti, Wesley Aoyama, Tatsuya Wilcox, Ethan Schneider, Nathan |
| contents | Humans have a remarkable ability to acquire and understand grammatical phenomena that are seen rarely, if ever, during childhood. Recent evidence suggests that language models with human-scale pretraining data may possess a similar ability by generalizing from frequent to rare constructions. However, it remains an open question how widespread this generalization ability is, and to what extent this knowledge extends to meanings of rare constructions, as opposed to just their forms. We fill this gap by testing human-scale transformer language models on their knowledge of both the form and meaning of the (rare and quirky) English LET-ALONE construction. To evaluate our LMs we construct a bespoke synthetic benchmark that targets syntactic and semantic properties of the construction. We find that human-scale LMs are sensitive to form, even when related constructions are filtered from the dataset. However, human-scale LMs do not make correct generalizations about LET-ALONE's meaning. These results point to an asymmetry in the current architectures' sample efficiency between language form and meaning, something which is not present in human language learners. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_04408 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaning Scivetti, Wesley Aoyama, Tatsuya Wilcox, Ethan Schneider, Nathan Computation and Language Artificial Intelligence Humans have a remarkable ability to acquire and understand grammatical phenomena that are seen rarely, if ever, during childhood. Recent evidence suggests that language models with human-scale pretraining data may possess a similar ability by generalizing from frequent to rare constructions. However, it remains an open question how widespread this generalization ability is, and to what extent this knowledge extends to meanings of rare constructions, as opposed to just their forms. We fill this gap by testing human-scale transformer language models on their knowledge of both the form and meaning of the (rare and quirky) English LET-ALONE construction. To evaluate our LMs we construct a bespoke synthetic benchmark that targets syntactic and semantic properties of the construction. We find that human-scale LMs are sensitive to form, even when related constructions are filtered from the dataset. However, human-scale LMs do not make correct generalizations about LET-ALONE's meaning. These results point to an asymmetry in the current architectures' sample efficiency between language form and meaning, something which is not present in human language learners. |
| title | Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.04408 |