Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917041214062592 |
|---|---|
| author | Nayeem, Mir Tafseer Alqahtani, Sawsan Laskar, Md Tahmid Rahman Mohiuddin, Tasnim Bari, M Saiful |
| author_facet | Nayeem, Mir Tafseer Alqahtani, Sawsan Laskar, Md Tahmid Rahman Mohiuddin, Tasnim Bari, M Saiful |
| contents | Tokenization is a crucial but under-evaluated step in large language models (LLMs). The standard metric, fertility (the average number of tokens per word), captures compression efficiency but obscures how vocabularies are allocated across languages and domains. We analyze six widely used tokenizers across seven languages and two domains, finding stable fertility for English, high fertility for Chinese, and little domain sensitivity. To address fertility's blind spots, we propose the Single Token Retention Rate (STRR), which measures the proportion of words preserved as single tokens. STRR reveals systematic prioritization of English, strong support for Chinese, and fragmentation in Hindi, offering an interpretable view of cross-lingual fairness. Our results show that STRR complements fertility and provides practical guidance for designing more equitable multilingual tokenizers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09947 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation Nayeem, Mir Tafseer Alqahtani, Sawsan Laskar, Md Tahmid Rahman Mohiuddin, Tasnim Bari, M Saiful Computation and Language Artificial Intelligence Machine Learning Tokenization is a crucial but under-evaluated step in large language models (LLMs). The standard metric, fertility (the average number of tokens per word), captures compression efficiency but obscures how vocabularies are allocated across languages and domains. We analyze six widely used tokenizers across seven languages and two domains, finding stable fertility for English, high fertility for Chinese, and little domain sensitivity. To address fertility's blind spots, we propose the Single Token Retention Rate (STRR), which measures the proportion of words preserved as single tokens. STRR reveals systematic prioritization of English, strong support for Chinese, and fragmentation in Hindi, offering an interpretable view of cross-lingual fairness. Our results show that STRR complements fertility and provides practical guidance for designing more equitable multilingual tokenizers. |
| title | Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2510.09947 |