What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915097945833472 |
|---|---|
| author | Borenstein, Nadav Svete, Anej Chan, Robin Valvoda, Josef Nowak, Franz Augenstein, Isabelle Chodroff, Eleanor Cotterell, Ryan |
| author_facet | Borenstein, Nadav Svete, Anej Chan, Robin Valvoda, Josef Nowak, Franz Augenstein, Isabelle Chodroff, Eleanor Cotterell, Ryan |
| contents | What can large language models learn? By definition, language models (LM) are distributions over strings. Therefore, an intuitive way of addressing the above question is to formalize it as a matter of learnability of classes of distributions over strings. While prior work in this direction focused on assessing the theoretical limits, in contrast, we seek to understand the empirical learnability. Unlike prior empirical work, we evaluate neural LMs on their home turf-learning probabilistic languages-rather than as classifiers of formal languages. In particular, we investigate the learnability of regular LMs (RLMs) by RNN and Transformer LMs. We empirically test the learnability of RLMs as a function of various complexity parameters of the RLM and the hidden state size of the neural LM. We find that the RLM rank, which corresponds to the size of linear space spanned by the logits of its conditional distributions, and the expected length of sampled strings are strong and significant predictors of learnability for both RNNs and Transformers. Several other predictors also reach significance, but with differing patterns between RNNs and Transformers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_04289 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages Borenstein, Nadav Svete, Anej Chan, Robin Valvoda, Josef Nowak, Franz Augenstein, Isabelle Chodroff, Eleanor Cotterell, Ryan Computation and Language What can large language models learn? By definition, language models (LM) are distributions over strings. Therefore, an intuitive way of addressing the above question is to formalize it as a matter of learnability of classes of distributions over strings. While prior work in this direction focused on assessing the theoretical limits, in contrast, we seek to understand the empirical learnability. Unlike prior empirical work, we evaluate neural LMs on their home turf-learning probabilistic languages-rather than as classifiers of formal languages. In particular, we investigate the learnability of regular LMs (RLMs) by RNN and Transformer LMs. We empirically test the learnability of RLMs as a function of various complexity parameters of the RLM and the hidden state size of the neural LM. We find that the RLM rank, which corresponds to the size of linear space spanned by the logits of its conditional distributions, and the expected length of sampled strings are strong and significant predictors of learnability for both RNNs and Transformers. Several other predictors also reach significance, but with differing patterns between RNNs and Transformers. |
| title | What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.04289 |