Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909310016028672 |
|---|---|
| author | Ruzzetti, Elena Sofia Ranaldi, Federico Logozzo, Felicia Mastromattei, Michele Ranaldi, Leonardo Zanzotto, Fabio Massimo |
| author_facet | Ruzzetti, Elena Sofia Ranaldi, Federico Logozzo, Felicia Mastromattei, Michele Ranaldi, Leonardo Zanzotto, Fabio Massimo |
| contents | The impressive achievements of transformers force NLP researchers to delve into how these models represent the underlying structure of natural language. In this paper, we propose a novel standpoint to investigate the above issue: using typological similarities among languages to observe how their respective monolingual models encode structural information. We aim to layer-wise compare transformers for typologically similar languages to observe whether these similarities emerge for particular layers. For this investigation, we propose to use Centered Kernel Alignment to measure similarity among weight matrices. We found that syntactic typological similarity is consistent with the similarity between the weights in the middle layers, which are the pretrained BERT layers to which syntax encoding is generally attributed. Moreover, we observe that a domain adaptation on semantically equivalent texts enhances this similarity among weight matrices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_02215 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages Ruzzetti, Elena Sofia Ranaldi, Federico Logozzo, Felicia Mastromattei, Michele Ranaldi, Leonardo Zanzotto, Fabio Massimo Computation and Language Artificial Intelligence The impressive achievements of transformers force NLP researchers to delve into how these models represent the underlying structure of natural language. In this paper, we propose a novel standpoint to investigate the above issue: using typological similarities among languages to observe how their respective monolingual models encode structural information. We aim to layer-wise compare transformers for typologically similar languages to observe whether these similarities emerge for particular layers. For this investigation, we propose to use Centered Kernel Alignment to measure similarity among weight matrices. We found that syntactic typological similarity is consistent with the similarity between the weights in the middle layers, which are the pretrained BERT layers to which syntax encoding is generally attributed. Moreover, we observe that a domain adaptation on semantically equivalent texts enhances this similarity among weight matrices. |
| title | Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2305.02215 |