Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.01860 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916626369085440 |
|---|---|
| author | Opper, Mattia Siddharth, N. |
| author_facet | Opper, Mattia Siddharth, N. |
| contents | This paper presents two simple improvements to the Self-Structuring AutoEncoder (Self-StrAE). Firstly, we show that including reconstruction to the vocabulary as an auxiliary objective improves representation quality. Secondly, we demonstrate that increasing the number of independent channels leads to significant improvements in embedding quality, while simultaneously reducing the number of parameters. Surprisingly, we demonstrate that this trend can be followed to the extreme, even to point of reducing the total number of non-embedding parameters to seven. Our system can be pre-trained from scratch with as little as 10M tokens of input data, and proves effective across English, Spanish and Afrikaans. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01860 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders Learn More With Less Opper, Mattia Siddharth, N. Computation and Language This paper presents two simple improvements to the Self-Structuring AutoEncoder (Self-StrAE). Firstly, we show that including reconstruction to the vocabulary as an auxiliary objective improves representation quality. Secondly, we demonstrate that increasing the number of independent channels leads to significant improvements in embedding quality, while simultaneously reducing the number of parameters. Surprisingly, we demonstrate that this trend can be followed to the extreme, even to point of reducing the total number of non-embedding parameters to seven. Our system can be pre-trained from scratch with as little as 10M tokens of input data, and proves effective across English, Spanish and Afrikaans. |
| title | Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders Learn More With Less |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.01860 |