Splintering Nonconcatenative Languages for Better Tokenization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910981822611456 |
|---|---|
| author | Gazit, Bar Shmidman, Shaltiel Shmidman, Avi Pinter, Yuval |
| author_facet | Gazit, Bar Shmidman, Shaltiel Shmidman, Avi Pinter, Yuval |
| contents | Common subword tokenization algorithms like BPE and UnigramLM assume that text can be split into meaningful units by concatenative measures alone. This is not true for languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. We present SPLINTER, a pre-processing step which rearranges text into a linear form that better represents such nonconcatenative morphologies, enabling meaningful contiguous segments to be found by the tokenizer. We demonstrate SPLINTER's merit using both intrinsic measures evaluating token vocabularies in Hebrew, Arabic, and Malay; as well as on downstream tasks using BERT-architecture models trained for Hebrew. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_14433 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Splintering Nonconcatenative Languages for Better Tokenization Gazit, Bar Shmidman, Shaltiel Shmidman, Avi Pinter, Yuval Computation and Language Common subword tokenization algorithms like BPE and UnigramLM assume that text can be split into meaningful units by concatenative measures alone. This is not true for languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. We present SPLINTER, a pre-processing step which rearranges text into a linear form that better represents such nonconcatenative morphologies, enabling meaningful contiguous segments to be found by the tokenizer. We demonstrate SPLINTER's merit using both intrinsic measures evaluating token vocabularies in Hebrew, Arabic, and Malay; as well as on downstream tasks using BERT-architecture models trained for Hebrew. |
| title | Splintering Nonconcatenative Languages for Better Tokenization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.14433 |