Vocabulary shapes cross-lingual variation of word-order learnability in language models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912975268347904 |
|---|---|
| author | Martins, Jonas Mayer Jumelet, Jaap Priesemann, Viola Beinborn, Lisa |
| author_facet | Martins, Jonas Mayer Jumelet, Jaap Priesemann, Viola Beinborn, Lisa |
| contents | Why do some languages like Czech permit free word order, while others like English do not? We address this question by pretraining transformer language models on a spectrum of synthetic word-order variants of natural languages. We observe that greater word-order irregularity consistently raises model surprisal, indicating reduced learnability. Sentence reversal, however, affects learnability only weakly. A coarse distinction of free- (e.g., Czech and Finnish) and fixed-word-order languages (e.g., English and French) does not explain cross-lingual variation. Instead, the structure of the word and subword vocabulary strongly predicts the model surprisal. Overall, vocabulary structure emerges as a key driver of computational word-order learnability across languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_19427 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Vocabulary shapes cross-lingual variation of word-order learnability in language models Martins, Jonas Mayer Jumelet, Jaap Priesemann, Viola Beinborn, Lisa Computation and Language Artificial Intelligence Machine Learning Why do some languages like Czech permit free word order, while others like English do not? We address this question by pretraining transformer language models on a spectrum of synthetic word-order variants of natural languages. We observe that greater word-order irregularity consistently raises model surprisal, indicating reduced learnability. Sentence reversal, however, affects learnability only weakly. A coarse distinction of free- (e.g., Czech and Finnish) and fixed-word-order languages (e.g., English and French) does not explain cross-lingual variation. Instead, the structure of the word and subword vocabulary strongly predicts the model surprisal. Overall, vocabulary structure emerges as a key driver of computational word-order learnability across languages. |
| title | Vocabulary shapes cross-lingual variation of word-order learnability in language models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2603.19427 |