The Token Tax: Systematic Bias in Multilingual Tokenization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908907687903232 |
|---|---|
| author | Lundin, Jessica M. Zhang, Ada Karim, Nihal Louzan, Hamza Wei, Victor Adelani, David Carroll, Cody |
| author_facet | Lundin, Jessica M. Zhang, Ada Karim, Nihal Louzan, Hamza Wei, Victor Adelani, David Carroll, Cody |
| contents | Tokenization inefficiency imposes structural disadvantages on morphologically complex, low-resource languages, inflating compute resources and depressing accuracy. We evaluate 10 large language models (LLMs) on AfriMMLU (9,000 MCQA items; 5 subjects; 16 African languages) and show that fertility (tokens/word) reliably predicts accuracy. Higher fertility consistently predicts lower accuracy across all models and subjects. We further find that reasoning models (DeepSeek, o1) consistently outperform non-reasoning peers across high and low resource languages in the AfriMMLU dataset, narrowing accuracy gaps observed in prior generations. Finally, translating token inflation to economics, a doubling in tokens results in quadrupled training cost and time, underscoring the token tax faced by many languages. These results motivate morphologically aware tokenization, fair pricing, and multilingual benchmarks for equitable natural language processing (NLP). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_05486 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Token Tax: Systematic Bias in Multilingual Tokenization Lundin, Jessica M. Zhang, Ada Karim, Nihal Louzan, Hamza Wei, Victor Adelani, David Carroll, Cody Computation and Language Artificial Intelligence Tokenization inefficiency imposes structural disadvantages on morphologically complex, low-resource languages, inflating compute resources and depressing accuracy. We evaluate 10 large language models (LLMs) on AfriMMLU (9,000 MCQA items; 5 subjects; 16 African languages) and show that fertility (tokens/word) reliably predicts accuracy. Higher fertility consistently predicts lower accuracy across all models and subjects. We further find that reasoning models (DeepSeek, o1) consistently outperform non-reasoning peers across high and low resource languages in the AfriMMLU dataset, narrowing accuracy gaps observed in prior generations. Finally, translating token inflation to economics, a doubling in tokens results in quadrupled training cost and time, underscoring the token tax faced by many languages. These results motivate morphologically aware tokenization, fair pricing, and multilingual benchmarks for equitable natural language processing (NLP). |
| title | The Token Tax: Systematic Bias in Multilingual Tokenization |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.05486 |