Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2021
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917351918665728 |
|---|---|
| author | Alqahtani, Sawsan Lalwani, Garima Zhang, Yi Romeo, Salvatore Mansour, Saab |
| author_facet | Alqahtani, Sawsan Lalwani, Garima Zhang, Yi Romeo, Salvatore Mansour, Saab |
| contents | Recent studies have proposed different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces. For contextualized embeddings, alignment becomes more complex as we additionally take context into consideration. In this work, we propose using Optimal Transport (OT) as an alignment objective during fine-tuning to further improve multilingual contextualized representations for downstream cross-lingual transfer. This approach does not require word-alignment pairs prior to fine-tuning that may lead to sub-optimal matching and instead learns the word alignments within context in an unsupervised manner. It also allows different types of mappings due to soft matching between source and target sentences. We benchmark our proposed method on two tasks (XNLI and XQuAD) and achieve improvements over baselines as well as competitive results compared to similar recent works. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2110_02887 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings Alqahtani, Sawsan Lalwani, Garima Zhang, Yi Romeo, Salvatore Mansour, Saab Computation and Language Artificial Intelligence Recent studies have proposed different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces. For contextualized embeddings, alignment becomes more complex as we additionally take context into consideration. In this work, we propose using Optimal Transport (OT) as an alignment objective during fine-tuning to further improve multilingual contextualized representations for downstream cross-lingual transfer. This approach does not require word-alignment pairs prior to fine-tuning that may lead to sub-optimal matching and instead learns the word alignments within context in an unsupervised manner. It also allows different types of mappings due to soft matching between source and target sentences. We benchmark our proposed method on two tasks (XNLI and XQuAD) and achieve improvements over baselines as well as competitive results compared to similar recent works. |
| title | Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2110.02887 |