Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.11380 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915292270034944 |
|---|---|
| author | Liu, Zhu Liu, Ying Luo, KangYang Kong, Cunliang Sun, Maosong |
| author_facet | Liu, Zhu Liu, Ying Luo, KangYang Kong, Cunliang Sun, Maosong |
| contents | Lexico-semantic networks represent words as nodes and their semantic relatedness as edges. While such networks are traditionally constructed using embeddings from encoder-based models or static vectors, embeddings from decoder-only large language models (LLMs) remain underexplored. Unlike encoder models, LLMs are trained with a next-token prediction objective, which does not directly encode the meaning of the current token. In this paper, we construct lexico-semantic networks from the input embeddings of LLMs with varying parameter scales and conduct a comparative analysis of their global and local structures. Our results show that these networks exhibit small-world properties, characterized by high clustering and short path lengths. Moreover, larger LLMs yield more intricate networks with less small-world effects and longer paths, reflecting richer semantic structures and relations. We further validate our approach through analyses of common conceptual pairs, structured lexical relations derived from WordNet, and a cross-lingual semantic network for qualitative words. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11380 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From the New World of Word Embeddings: A Comparative Study of Small-World Lexico-Semantic Networks in LLMs Liu, Zhu Liu, Ying Luo, KangYang Kong, Cunliang Sun, Maosong Computation and Language Lexico-semantic networks represent words as nodes and their semantic relatedness as edges. While such networks are traditionally constructed using embeddings from encoder-based models or static vectors, embeddings from decoder-only large language models (LLMs) remain underexplored. Unlike encoder models, LLMs are trained with a next-token prediction objective, which does not directly encode the meaning of the current token. In this paper, we construct lexico-semantic networks from the input embeddings of LLMs with varying parameter scales and conduct a comparative analysis of their global and local structures. Our results show that these networks exhibit small-world properties, characterized by high clustering and short path lengths. Moreover, larger LLMs yield more intricate networks with less small-world effects and longer paths, reflecting richer semantic structures and relations. We further validate our approach through analyses of common conceptual pairs, structured lexical relations derived from WordNet, and a cross-lingual semantic network for qualitative words. |
| title | From the New World of Word Embeddings: A Comparative Study of Small-World Lexico-Semantic Networks in LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.11380 |