PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914160567123968 |
|---|---|
| author | Sriwirote, Panyut Thapiang, Jalinee Timtong, Vasan Rutherford, Attapol T. |
| author_facet | Sriwirote, Panyut Thapiang, Jalinee Timtong, Vasan Rutherford, Attapol T. |
| contents | While WangchanBERTa has become the de facto standard in transformer-based Thai language modeling, it still has shortcomings in regard to the understanding of foreign words, most notably English words, which are often borrowed without orthographic assimilation into Thai in many contexts. We identify the lack of foreign vocabulary in WangchanBERTa's tokenizer as the main source of these shortcomings. We then expand WangchanBERTa's vocabulary via vocabulary transfer from XLM-R's pretrained tokenizer and pretrain a new model using the expanded tokenizer, starting from WangchanBERTa's checkpoint, on a new dataset that is larger than the one used to train WangchanBERTa. Our results show that our new pretrained model, PhayaThaiBERT, outperforms WangchanBERTa in many downstream tasks and datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_12475 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords Sriwirote, Panyut Thapiang, Jalinee Timtong, Vasan Rutherford, Attapol T. Computation and Language Artificial Intelligence While WangchanBERTa has become the de facto standard in transformer-based Thai language modeling, it still has shortcomings in regard to the understanding of foreign words, most notably English words, which are often borrowed without orthographic assimilation into Thai in many contexts. We identify the lack of foreign vocabulary in WangchanBERTa's tokenizer as the main source of these shortcomings. We then expand WangchanBERTa's vocabulary via vocabulary transfer from XLM-R's pretrained tokenizer and pretrain a new model using the expanded tokenizer, starting from WangchanBERTa's checkpoint, on a new dataset that is larger than the one used to train WangchanBERTa. Our results show that our new pretrained model, PhayaThaiBERT, outperforms WangchanBERTa in many downstream tasks and datasets. |
| title | PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2311.12475 |