Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.23510 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910678851256320 |
|---|---|
| author | Belcak, Peter Wattenhofer, Roger |
| author_facet | Belcak, Peter Wattenhofer, Roger |
| contents | It is staggering that words of the English language, which are on average represented by 5--6 bytes of ASCII, require as much as 24 kilobytes when served to large language models. We show that there is room for more information in every token embedding. We demonstrate that 1--3-layer transformers are capable of encoding and subsequently decoding standard English sentences into as little as a single 3-kilobyte token. Our work implies that even small networks can learn to construct valid English sentences and suggests the possibility of optimising large language models by moving from sub-word token embeddings towards larger fragments of text. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23510 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Tiny Transformers Excel at Sentence Compression Belcak, Peter Wattenhofer, Roger Machine Learning Computation and Language It is staggering that words of the English language, which are on average represented by 5--6 bytes of ASCII, require as much as 24 kilobytes when served to large language models. We show that there is room for more information in every token embedding. We demonstrate that 1--3-layer transformers are capable of encoding and subsequently decoding standard English sentences into as little as a single 3-kilobyte token. Our work implies that even small networks can learn to construct valid English sentences and suggests the possibility of optimising large language models by moving from sub-word token embeddings towards larger fragments of text. |
| title | Tiny Transformers Excel at Sentence Compression |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2410.23510 |