A Morphology-Based Investigation of Positional Encodings
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914816265814016 |
|---|---|
| author | Ghosh, Poulami Vashishth, Shikhar Dabre, Raj Bhattacharyya, Pushpak |
| author_facet | Ghosh, Poulami Vashishth, Shikhar Dabre, Raj Bhattacharyya, Pushpak |
| contents | Contemporary deep learning models effectively handle languages with diverse morphology despite not being directly integrated into them. Morphology and word order are closely linked, with the latter incorporated into transformer-based models through positional encodings. This prompts a fundamental inquiry: Is there a correlation between the morphological complexity of a language and the utilization of positional encoding in pre-trained language models? In pursuit of an answer, we present the first study addressing this question, encompassing 22 languages and 5 downstream tasks. Our findings reveal that the importance of positional encoding diminishes with increasing morphological complexity in languages. Our study motivates the need for a deeper understanding of positional encoding, augmenting them to better reflect the different languages under consideration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04530 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Morphology-Based Investigation of Positional Encodings Ghosh, Poulami Vashishth, Shikhar Dabre, Raj Bhattacharyya, Pushpak Computation and Language Contemporary deep learning models effectively handle languages with diverse morphology despite not being directly integrated into them. Morphology and word order are closely linked, with the latter incorporated into transformer-based models through positional encodings. This prompts a fundamental inquiry: Is there a correlation between the morphological complexity of a language and the utilization of positional encoding in pre-trained language models? In pursuit of an answer, we present the first study addressing this question, encompassing 22 languages and 5 downstream tasks. Our findings reveal that the importance of positional encoding diminishes with increasing morphological complexity in languages. Our study motivates the need for a deeper understanding of positional encoding, augmenting them to better reflect the different languages under consideration. |
| title | A Morphology-Based Investigation of Positional Encodings |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.04530 |