Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.09863 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909851362263040 |
|---|---|
| author | Karkada, Dhruva Simon, James B. Bahri, Yasaman DeWeese, Michael R. |
| author_facet | Karkada, Dhruva Simon, James B. Bahri, Yasaman DeWeese, Michael R. |
| contents | Self-supervised word embedding algorithms such as word2vec provide a minimal setting for studying representation learning in language modeling. We examine the quartic Taylor approximation of the word2vec loss around the origin, and we show that both the resulting training dynamics and the final performance on downstream tasks are empirically very similar to those of word2vec. Our main contribution is to analytically solve for both the gradient flow training dynamics and the final word embeddings in terms of only the corpus statistics and training hyperparameters. The solutions reveal that these models learn orthogonal linear subspaces one at a time, each one incrementing the effective rank of the embeddings until model capacity is saturated. Training on Wikipedia, we find that each of the top linear subspaces represents an interpretable topic-level concept. Finally, we apply our theory to describe how linear representations of more abstract semantic concepts emerge during training; these can be used to complete analogies via vector addition. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_09863 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models Karkada, Dhruva Simon, James B. Bahri, Yasaman DeWeese, Michael R. Machine Learning Computation and Language Self-supervised word embedding algorithms such as word2vec provide a minimal setting for studying representation learning in language modeling. We examine the quartic Taylor approximation of the word2vec loss around the origin, and we show that both the resulting training dynamics and the final performance on downstream tasks are empirically very similar to those of word2vec. Our main contribution is to analytically solve for both the gradient flow training dynamics and the final word embeddings in terms of only the corpus statistics and training hyperparameters. The solutions reveal that these models learn orthogonal linear subspaces one at a time, each one incrementing the effective rank of the embeddings until model capacity is saturated. Training on Wikipedia, we find that each of the top linear subspaces represents an interpretable topic-level concept. Finally, we apply our theory to describe how linear representations of more abstract semantic concepts emerge during training; these can be used to complete analogies via vector addition. |
| title | Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.09863 |