Greed is All You Need: An Evaluation of Tokenizer Inference Methods
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929367611867136 |
|---|---|
| author | Uzan, Omri Schmidt, Craig W. Tanner, Chris Pinter, Yuval |
| author_facet | Uzan, Omri Schmidt, Craig W. Tanner, Chris Pinter, Yuval |
| contents | While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabularies is often left unspecified, or ill-suited to the method in which they were constructed. We provide a controlled analysis of seven tokenizer inference methods across four different algorithms and three vocabulary sizes, performed on a novel intrinsic evaluation suite we curated for English, combining measures rooted in morphology, cognition, and information theory. We show that for the most commonly used tokenizers, greedy inference performs surprisingly well; and that SaGe, a recently-introduced contextually-informed tokenizer, outperforms all others on morphological alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01289 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Greed is All You Need: An Evaluation of Tokenizer Inference Methods Uzan, Omri Schmidt, Craig W. Tanner, Chris Pinter, Yuval Computation and Language While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabularies is often left unspecified, or ill-suited to the method in which they were constructed. We provide a controlled analysis of seven tokenizer inference methods across four different algorithms and three vocabulary sizes, performed on a novel intrinsic evaluation suite we curated for English, combining measures rooted in morphology, cognition, and information theory. We show that for the most commonly used tokenizers, greedy inference performs surprisingly well; and that SaGe, a recently-introduced contextually-informed tokenizer, outperforms all others on morphological alignment. |
| title | Greed is All You Need: An Evaluation of Tokenizer Inference Methods |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.01289 |