Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.18072 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910484474626048 |
|---|---|
| author | Luitel, Nishant Bekoju, Nirajan Sah, Anand Kumar Shakya, Subarna |
| author_facet | Luitel, Nishant Bekoju, Nirajan Sah, Anand Kumar Shakya, Subarna |
| contents | The task of Spell Correction(SC) in low-resource languages presents a significant challenge due to the availability of only a limited corpus of data and no annotated spelling correction datasets. To tackle these challenges a small-scale word-based transformer LM is trained to provide the SC model with contextual understanding. Further, the probabilistic error rules are extracted from the corpus in an unsupervised way to model the tendency of error happening(error model). Then the combination of LM and error model is used to develop the SC model through the well-known noisy channel framework. The effectiveness of this approach is demonstrated through experiments on the Nepali language where there is access to just an unprocessed corpus of textual data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_18072 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Contextual Spelling Correction with Language Model for Low-resource Setting Luitel, Nishant Bekoju, Nirajan Sah, Anand Kumar Shakya, Subarna Computation and Language The task of Spell Correction(SC) in low-resource languages presents a significant challenge due to the availability of only a limited corpus of data and no annotated spelling correction datasets. To tackle these challenges a small-scale word-based transformer LM is trained to provide the SC model with contextual understanding. Further, the probabilistic error rules are extracted from the corpus in an unsupervised way to model the tendency of error happening(error model). Then the combination of LM and error model is used to develop the SC model through the well-known noisy channel framework. The effectiveness of this approach is demonstrated through experiments on the Nepali language where there is access to just an unprocessed corpus of textual data. |
| title | Contextual Spelling Correction with Language Model for Low-resource Setting |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.18072 |