Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916911704440832 |
|---|---|
| author | Foroutan, Negar Meister, Clara Paul, Debjit Niklaus, Joel Ahmadi, Sina Bosselut, Antoine Sennrich, Rico |
| author_facet | Foroutan, Negar Meister, Clara Paul, Debjit Niklaus, Joel Ahmadi, Sina Bosselut, Antoine Sennrich, Rico |
| contents | Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines. Standard algorithms for learning tokenizers rely on frequency-based objectives, which favor languages dominant in the training data and consequently leave lower-resource languages with tokenizations that are disproportionately longer, morphologically implausible, or even riddled with <UNK> placeholders. This phenomenon ultimately amplifies computational and financial inequalities between users from different language backgrounds. To remedy this, we introduce Parity-aware Byte Pair Encoding (BPE), a variant of the widely-used BPE algorithm. At every merge step, Parity-aware BPE maximizes the compression gain of the currently worst-compressed language, trading a small amount of global compression for cross-lingual parity. We find empirically that Parity-aware BPE leads to more equitable token counts across languages, with negligible impact on global compression rate and no substantial effect on language-model performance in downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_04796 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization Foroutan, Negar Meister, Clara Paul, Debjit Niklaus, Joel Ahmadi, Sina Bosselut, Antoine Sennrich, Rico Computation and Language Artificial Intelligence Machine Learning Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines. Standard algorithms for learning tokenizers rely on frequency-based objectives, which favor languages dominant in the training data and consequently leave lower-resource languages with tokenizations that are disproportionately longer, morphologically implausible, or even riddled with <UNK> placeholders. This phenomenon ultimately amplifies computational and financial inequalities between users from different language backgrounds. To remedy this, we introduce Parity-aware Byte Pair Encoding (BPE), a variant of the widely-used BPE algorithm. At every merge step, Parity-aware BPE maximizes the compression gain of the currently worst-compressed language, trading a small amount of global compression for cross-lingual parity. We find empirically that Parity-aware BPE leads to more equitable token counts across languages, with negligible impact on global compression rate and no substantial effect on language-model performance in downstream tasks. |
| title | Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2508.04796 |