Understanding Higher-Order Correlations Among Semantic Components in Embeddings
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929532947136512 |
|---|---|
| author | Oyama, Momose Yamagiwa, Hiroaki Shimodaira, Hidetoshi |
| author_facet | Oyama, Momose Yamagiwa, Hiroaki Shimodaira, Hidetoshi |
| contents | Independent Component Analysis (ICA) offers interpretable semantic components of embeddings. While ICA theory assumes that embeddings can be linearly decomposed into independent components, real-world data often do not satisfy this assumption. Consequently, non-independencies remain between the estimated components, which ICA cannot eliminate. We quantified these non-independencies using higher-order correlations and demonstrated that when the higher-order correlation between two components is large, it indicates a strong semantic association between them, along with many words sharing common meanings with both components. The entire structure of non-independencies was visualized using a maximum spanning tree of semantic components. These findings provide deeper insights into embeddings through ICA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_19919 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding Higher-Order Correlations Among Semantic Components in Embeddings Oyama, Momose Yamagiwa, Hiroaki Shimodaira, Hidetoshi Computation and Language Independent Component Analysis (ICA) offers interpretable semantic components of embeddings. While ICA theory assumes that embeddings can be linearly decomposed into independent components, real-world data often do not satisfy this assumption. Consequently, non-independencies remain between the estimated components, which ICA cannot eliminate. We quantified these non-independencies using higher-order correlations and demonstrated that when the higher-order correlation between two components is large, it indicates a strong semantic association between them, along with many words sharing common meanings with both components. The entire structure of non-independencies was visualized using a maximum spanning tree of semantic components. These findings provide deeper insights into embeddings through ICA. |
| title | Understanding Higher-Order Correlations Among Semantic Components in Embeddings |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.19919 |