Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2601.18030 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915755348459520 |
|---|---|
| author | Rabe, Markus N. Clymo, Judith Dong, Zheren |
| author_facet | Rabe, Markus N. Clymo, Judith Dong, Zheren |
| contents | We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on spelling, but also across standard benchmarks. We conduct scaling studies for models with 40M to 800M parameters, which suggest that the improvements are equivalent to needing about 8% less compute and data to achieve the same test loss. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_18030 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Spelling Bee Embeddings for Language Modeling Rabe, Markus N. Clymo, Judith Dong, Zheren Machine Learning We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on spelling, but also across standard benchmarks. We conduct scaling studies for models with 40M to 800M parameters, which suggest that the improvements are equivalent to needing about 8% less compute and data to achieve the same test loss. |
| title | Spelling Bee Embeddings for Language Modeling |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.18030 |