Two Counterexamples to Tokenization and the Noiseless Channel
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929259075862528 |
|---|---|
| author | Cognetta, Marco Zouhar, Vilém Moon, Sangwhan Okazaki, Naoaki |
| author_facet | Cognetta, Marco Zouhar, Vilém Moon, Sangwhan Okazaki, Naoaki |
| contents | In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), Rényi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest Rényi efficiency of the unigram distribution should be chosen. The Rényi efficiency is thus treated as a predictor of downstream performance (e.g., predicting BLEU for a machine translation task), without the expensive step of training multiple models with different tokenizers. Although useful, the predictive power of this metric is not perfect, and the authors note there are additional qualities of a good tokenization scheme that Rényi efficiency alone cannot capture.
We describe two variants of BPE tokenization which can arbitrarily increase Rényi efficiency while decreasing the downstream model performance. These counterexamples expose cases where Rényi efficiency fails as an intrinsic tokenization metric and thus give insight for building more accurate predictors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_14614 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Two Counterexamples to Tokenization and the Noiseless Channel Cognetta, Marco Zouhar, Vilém Moon, Sangwhan Okazaki, Naoaki Computation and Language In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), Rényi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest Rényi efficiency of the unigram distribution should be chosen. The Rényi efficiency is thus treated as a predictor of downstream performance (e.g., predicting BLEU for a machine translation task), without the expensive step of training multiple models with different tokenizers. Although useful, the predictive power of this metric is not perfect, and the authors note there are additional qualities of a good tokenization scheme that Rényi efficiency alone cannot capture. We describe two variants of BPE tokenization which can arbitrarily increase Rényi efficiency while decreasing the downstream model performance. These counterexamples expose cases where Rényi efficiency fails as an intrinsic tokenization metric and thus give insight for building more accurate predictors. |
| title | Two Counterexamples to Tokenization and the Noiseless Channel |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.14614 |