Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.03194 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909335846649856 |
|---|---|
| author | Kumari, Vibhuti Kavi, Narayana Murthy |
| author_facet | Kumari, Vibhuti Kavi, Narayana Murthy |
| contents | In this paper we propose a novel method of augmenting parallel text corpora which promises good quality and is also capable of producing many fold larger corpora than the seed corpus we start with. We do not need any additional monolingual corpora. We use Multi-Lingual Masked Language Model to mask and predict alternative words in context and we use Sentence Embeddings to check and select sentence pairs which are likely to be translations of each other. We cross check our method using metrics for MT Quality Estimation. We believe this method can greatly alleviate the data scarcity problem for all language pairs for which a reasonable seed corpus is available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03194 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Parallel Corpus Augmentation using Masked Language Models Kumari, Vibhuti Kavi, Narayana Murthy Computation and Language In this paper we propose a novel method of augmenting parallel text corpora which promises good quality and is also capable of producing many fold larger corpora than the seed corpus we start with. We do not need any additional monolingual corpora. We use Multi-Lingual Masked Language Model to mask and predict alternative words in context and we use Sentence Embeddings to check and select sentence pairs which are likely to be translations of each other. We cross check our method using metrics for MT Quality Estimation. We believe this method can greatly alleviate the data scarcity problem for all language pairs for which a reasonable seed corpus is available. |
| title | Parallel Corpus Augmentation using Masked Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.03194 |