Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.18762 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915408548724736 |
|---|---|
| author | Abdullah, Abdulhady Abas Gandomi, Amir H. Rashid, Tarik A Mirjalili, Seyedali Abualigah, Laith Živković, Milena Veisi, Hadi |
| author_facet | Abdullah, Abdulhady Abas Gandomi, Amir H. Rashid, Tarik A Mirjalili, Seyedali Abualigah, Laith Živković, Milena Veisi, Hadi |
| contents | In natural language processing, multilingual models like mBERT and XLM-RoBERTa promise broad coverage but often struggle with languages that share a script yet differ in orthographic norms and cultural context. This issue is especially notable in Arabic-script languages such as Kurdish Sorani, Arabic, Persian, and Urdu. We introduce the Arabic Script RoBERTa (AS-RoBERTa) family: four RoBERTa-based models, each pre-trained on a large corpus tailored to its specific language. By focusing pre-training on language-specific script features and statistics, our models capture patterns overlooked by general-purpose models. When fine-tuned on classification tasks, AS-RoBERTa variants outperform mBERT and XLM-RoBERTa by 2 to 5 percentage points. An ablation study confirms that script-focused pre-training is central to these gains. Error analysis using confusion matrices shows how shared script traits and domain-specific content affect performance. Our results highlight the value of script-aware specialization for languages using the Arabic script and support further work on pre-training strategies rooted in script and language specificity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_18762 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Role of Orthographic Consistency in Multilingual Embedding Models for Text Classification in Arabic-Script Languages Abdullah, Abdulhady Abas Gandomi, Amir H. Rashid, Tarik A Mirjalili, Seyedali Abualigah, Laith Živković, Milena Veisi, Hadi Computation and Language In natural language processing, multilingual models like mBERT and XLM-RoBERTa promise broad coverage but often struggle with languages that share a script yet differ in orthographic norms and cultural context. This issue is especially notable in Arabic-script languages such as Kurdish Sorani, Arabic, Persian, and Urdu. We introduce the Arabic Script RoBERTa (AS-RoBERTa) family: four RoBERTa-based models, each pre-trained on a large corpus tailored to its specific language. By focusing pre-training on language-specific script features and statistics, our models capture patterns overlooked by general-purpose models. When fine-tuned on classification tasks, AS-RoBERTa variants outperform mBERT and XLM-RoBERTa by 2 to 5 percentage points. An ablation study confirms that script-focused pre-training is central to these gains. Error analysis using confusion matrices shows how shared script traits and domain-specific content affect performance. Our results highlight the value of script-aware specialization for languages using the Arabic script and support further work on pre-training strategies rooted in script and language specificity. |
| title | The Role of Orthographic Consistency in Multilingual Embedding Models for Text Classification in Arabic-Script Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.18762 |