AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910050229944320 |
|---|---|
| author | Elshehy, Omar Nacar, Omer Djamai, Abdelbasset Ragab, Muhammed Jallad, Khloud Al Abdelazim, Mona |
| author_facet | Elshehy, Omar Nacar, Omer Djamai, Abdelbasset Ragab, Muhammed Jallad, Khloud Al Abdelazim, Mona |
| contents | Encoder-only transformer models remain widely used for discriminative NLP tasks, yet recent architectural advances have largely focused on English. In this work, we present AraModernBERT, an adaptation of the ModernBERT encoder architecture to Arabic, and study the impact of transtokenized embedding initialization and native long-context modeling up to 8,192 tokens. We show that transtokenization is essential for Arabic language modeling, yielding dramatic improvements in masked language modeling performance compared to non-transtokenized initialization. We further demonstrate that AraModernBERT supports stable and effective long-context modeling, achieving improved intrinsic language modeling performance at extended sequence lengths. Downstream evaluations on Arabic natural language understanding tasks, including inference, offensive language detection, question-question similarity, and named entity recognition, confirm strong transfer to discriminative and sequence labeling settings. Our results highlight practical considerations for adapting modern encoder architectures to Arabic and other languages written in Arabic-derived scripts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_09982 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic Elshehy, Omar Nacar, Omer Djamai, Abdelbasset Ragab, Muhammed Jallad, Khloud Al Abdelazim, Mona Computation and Language Artificial Intelligence Encoder-only transformer models remain widely used for discriminative NLP tasks, yet recent architectural advances have largely focused on English. In this work, we present AraModernBERT, an adaptation of the ModernBERT encoder architecture to Arabic, and study the impact of transtokenized embedding initialization and native long-context modeling up to 8,192 tokens. We show that transtokenization is essential for Arabic language modeling, yielding dramatic improvements in masked language modeling performance compared to non-transtokenized initialization. We further demonstrate that AraModernBERT supports stable and effective long-context modeling, achieving improved intrinsic language modeling performance at extended sequence lengths. Downstream evaluations on Arabic natural language understanding tasks, including inference, offensive language detection, question-question similarity, and named entity recognition, confirm strong transfer to discriminative and sequence labeling settings. Our results highlight practical considerations for adapting modern encoder architectures to Arabic and other languages written in Arabic-derived scripts. |
| title | AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.09982 |