Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929546374152192 |
|---|---|
| author | Hayakawa, Daichi Sato, Issei |
| author_facet | Hayakawa, Daichi Sato, Issei |
| contents | In this study, we provide constructive proof that Transformers can recognize and generate hierarchical language efficiently with respect to model size, even without the need for a specific positional encoding. Specifically, we show that causal masking and a starting token enable Transformers to compute positional information and depth within hierarchical structures. We demonstrate that Transformers without positional encoding can generate hierarchical languages. Furthermore, we suggest that explicit positional encoding might have a detrimental effect on generalization with respect to sequence length. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12413 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding Hayakawa, Daichi Sato, Issei Computation and Language In this study, we provide constructive proof that Transformers can recognize and generate hierarchical language efficiently with respect to model size, even without the need for a specific positional encoding. Specifically, we show that causal masking and a starting token enable Transformers to compute positional information and depth within hierarchical structures. We demonstrate that Transformers without positional encoding can generate hierarchical languages. Furthermore, we suggest that explicit positional encoding might have a detrimental effect on generalization with respect to sequence length. |
| title | Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.12413 |