Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2305.17026 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929534472814592 |
|---|---|
| author | Roberts, Jesse |
| author_facet | Roberts, Jesse |
| contents | In this article we prove that the general transformer neural model undergirding modern large language models (LLMs) is Turing complete under reasonable assumptions. This is the first work to directly address the Turing completeness of the underlying technology employed in GPT-x as past work has focused on the more expressive, full auto-encoder transformer architecture. From this theoretical analysis, we show that the sparsity/compressibility of the word embedding is an important consideration for Turing completeness to hold. We also show that Transformers are are a variant of B machines studied by Hao Wang. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_17026 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | How Powerful are Decoder-Only Transformer Neural Models? Roberts, Jesse Computation and Language Machine Learning In this article we prove that the general transformer neural model undergirding modern large language models (LLMs) is Turing complete under reasonable assumptions. This is the first work to directly address the Turing completeness of the underlying technology employed in GPT-x as past work has focused on the more expressive, full auto-encoder transformer architecture. From this theoretical analysis, we show that the sparsity/compressibility of the word embedding is an important consideration for Turing completeness to hold. We also show that Transformers are are a variant of B machines studied by Hao Wang. |
| title | How Powerful are Decoder-Only Transformer Neural Models? |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2305.17026 |