On Self-improving Token Embeddings
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910915353378816 |
|---|---|
| author | Kubek, Mario M. Pokharel, Shiraj Böhme, Thomas McDaniel, Emma L. Unger, Herwig Mikler, Armin R. |
| author_facet | Kubek, Mario M. Pokharel, Shiraj Böhme, Thomas McDaniel, Emma L. Unger, Herwig Mikler, Armin R. |
| contents | This article introduces a novel and fast method for refining pre-trained static word or, more generally, token embeddings. By incorporating the embeddings of neighboring tokens in text corpora, it continuously updates the representation of each token, including those without pre-assigned embeddings. This approach effectively addresses the out-of-vocabulary problem, too. Operating independently of large language models and shallow neural networks, it enables versatile applications such as corpus exploration, conceptual search, and word sense disambiguation. The method is designed to enhance token representations within topically homogeneous corpora, where the vocabulary is restricted to a specific domain, resulting in more meaningful embeddings compared to general-purpose pre-trained vectors. As an example, the methodology is applied to explore storm events and their impacts on infrastructure and communities using narratives from a subset of the NOAA Storm Events database. The article also demonstrates how the approach improves the representation of storm-related terms over time, providing valuable insights into the evolving nature of disaster narratives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14808 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On Self-improving Token Embeddings Kubek, Mario M. Pokharel, Shiraj Böhme, Thomas McDaniel, Emma L. Unger, Herwig Mikler, Armin R. Computation and Language Artificial Intelligence Information Retrieval Machine Learning 68T50, 68T07 I.2.6; I.2.7; H.3.3 This article introduces a novel and fast method for refining pre-trained static word or, more generally, token embeddings. By incorporating the embeddings of neighboring tokens in text corpora, it continuously updates the representation of each token, including those without pre-assigned embeddings. This approach effectively addresses the out-of-vocabulary problem, too. Operating independently of large language models and shallow neural networks, it enables versatile applications such as corpus exploration, conceptual search, and word sense disambiguation. The method is designed to enhance token representations within topically homogeneous corpora, where the vocabulary is restricted to a specific domain, resulting in more meaningful embeddings compared to general-purpose pre-trained vectors. As an example, the methodology is applied to explore storm events and their impacts on infrastructure and communities using narratives from a subset of the NOAA Storm Events database. The article also demonstrates how the approach improves the representation of storm-related terms over time, providing valuable insights into the evolving nature of disaster narratives. |
| title | On Self-improving Token Embeddings |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning 68T50, 68T07 I.2.6; I.2.7; H.3.3 |
| url | https://arxiv.org/abs/2504.14808 |