Can Transformers Learn $n$-gram Language Models?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909335552000000 |
|---|---|
| author | Svete, Anej Borenstein, Nadav Zhou, Mike Augenstein, Isabelle Cotterell, Ryan |
| author_facet | Svete, Anej Borenstein, Nadav Zhou, Mike Augenstein, Isabelle Cotterell, Ryan |
| contents | Much theoretical work has described the ability of transformers to represent formal languages. However, linking theoretical results to empirical performance is not straightforward due to the complex interplay between the architecture, the learning algorithm, and training data. To test whether theoretical lower bounds imply \emph{learnability} of formal languages, we turn to recent work relating transformers to $n$-gram language models (LMs). We study transformers' ability to learn random $n$-gram LMs of two kinds: ones with arbitrary next-symbol probabilities and ones where those are defined with shared parameters. We find that classic estimation techniques for $n$-gram LMs such as add-$λ$ smoothing outperform transformers on the former, while transformers perform better on the latter, outperforming methods specifically designed to learn $n$-gram LMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03001 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can Transformers Learn $n$-gram Language Models? Svete, Anej Borenstein, Nadav Zhou, Mike Augenstein, Isabelle Cotterell, Ryan Computation and Language Much theoretical work has described the ability of transformers to represent formal languages. However, linking theoretical results to empirical performance is not straightforward due to the complex interplay between the architecture, the learning algorithm, and training data. To test whether theoretical lower bounds imply \emph{learnability} of formal languages, we turn to recent work relating transformers to $n$-gram language models (LMs). We study transformers' ability to learn random $n$-gram LMs of two kinds: ones with arbitrary next-symbol probabilities and ones where those are defined with shared parameters. We find that classic estimation techniques for $n$-gram LMs such as add-$λ$ smoothing outperform transformers on the former, while transformers perform better on the latter, outperforming methods specifically designed to learn $n$-gram LMs. |
| title | Can Transformers Learn $n$-gram Language Models? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.03001 |