Entropy2Vec: Crosslingual Language Modeling Entropy as End-to-End Learnable Language Representations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908521291841536 |
|---|---|
| author | Irawan, Patrick Amadeus Diandaru, Ryandito Syuhada, Belati Jagad Bintang Suchrady, Randy Zakya Aji, Alham Fikri Winata, Genta Indra Koto, Fajri Cahyawijaya, Samuel |
| author_facet | Irawan, Patrick Amadeus Diandaru, Ryandito Syuhada, Belati Jagad Bintang Suchrady, Randy Zakya Aji, Alham Fikri Winata, Genta Indra Koto, Fajri Cahyawijaya, Samuel |
| contents | We introduce Entropy2Vec, a novel framework for deriving cross-lingual language representations by leveraging the entropy of monolingual language models. Unlike traditional typological inventories that suffer from feature sparsity and static snapshots, Entropy2Vec uses the inherent uncertainty in language models to capture typological relationships between languages. By training a language model on a single language, we hypothesize that the entropy of its predictions reflects its structural similarity to other languages: Low entropy indicates high similarity, while high entropy suggests greater divergence. This approach yields dense, non-sparse language embeddings that are adaptable to different timeframes and free from missing values. Empirical evaluations demonstrate that Entropy2Vec embeddings align with established typological categories and achieved competitive performance in downstream multilingual NLP tasks, such as those addressed by the LinguAlchemy framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_05060 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Entropy2Vec: Crosslingual Language Modeling Entropy as End-to-End Learnable Language Representations Irawan, Patrick Amadeus Diandaru, Ryandito Syuhada, Belati Jagad Bintang Suchrady, Randy Zakya Aji, Alham Fikri Winata, Genta Indra Koto, Fajri Cahyawijaya, Samuel Computation and Language We introduce Entropy2Vec, a novel framework for deriving cross-lingual language representations by leveraging the entropy of monolingual language models. Unlike traditional typological inventories that suffer from feature sparsity and static snapshots, Entropy2Vec uses the inherent uncertainty in language models to capture typological relationships between languages. By training a language model on a single language, we hypothesize that the entropy of its predictions reflects its structural similarity to other languages: Low entropy indicates high similarity, while high entropy suggests greater divergence. This approach yields dense, non-sparse language embeddings that are adaptable to different timeframes and free from missing values. Empirical evaluations demonstrate that Entropy2Vec embeddings align with established typological categories and achieved competitive performance in downstream multilingual NLP tasks, such as those addressed by the LinguAlchemy framework. |
| title | Entropy2Vec: Crosslingual Language Modeling Entropy as End-to-End Learnable Language Representations |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.05060 |