Spelling Bee Embeddings for Language Modeling
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915755348459520 |
|---|---|
| author | Rabe, Markus N. Clymo, Judith Dong, Zheren |
| author_facet | Rabe, Markus N. Clymo, Judith Dong, Zheren |
| contents | We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on spelling, but also across standard benchmarks. We conduct scaling studies for models with 40M to 800M parameters, which suggest that the improvements are equivalent to needing about 8% less compute and data to achieve the same test loss. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_18030 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Spelling Bee Embeddings for Language Modeling Rabe, Markus N. Clymo, Judith Dong, Zheren Machine Learning We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on spelling, but also across standard benchmarks. We conduct scaling studies for models with 40M to 800M parameters, which suggest that the improvements are equivalent to needing about 8% less compute and data to achieve the same test loss. |
| title | Spelling Bee Embeddings for Language Modeling |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.18030 |