Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.11518 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914259425820672 |
|---|---|
| author | Roberts, Jonathan Han, Kai Albanie, Samuel |
| author_facet | Roberts, Jonathan Han, Kai Albanie, Samuel |
| contents | Frontier LLMs are increasingly utilised across academia, society and industry. A commonly used unit for comparing models, their inputs and outputs, and estimating inference pricing is the token. In general, tokens are used as a stable currency, assumed to be broadly consistent across tokenizers and contexts, enabling direct comparisons. However, tokenization varies significantly across models and domains of text, making naive interpretation of token counts problematic. We quantify this variation by providing a comprehensive empirical analysis of tokenization, exploring the compression of sequences to tokens across different distributions of textual data. Our analysis challenges commonly held heuristics about token lengths, finding them to be overly simplistic. We hope the insights of our study add clarity and intuition toward tokenization in contemporary LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_11518 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers Roberts, Jonathan Han, Kai Albanie, Samuel Computation and Language Frontier LLMs are increasingly utilised across academia, society and industry. A commonly used unit for comparing models, their inputs and outputs, and estimating inference pricing is the token. In general, tokens are used as a stable currency, assumed to be broadly consistent across tokenizers and contexts, enabling direct comparisons. However, tokenization varies significantly across models and domains of text, making naive interpretation of token counts problematic. We quantify this variation by providing a comprehensive empirical analysis of tokenization, exploring the compression of sequences to tokens across different distributions of textual data. Our analysis challenges commonly held heuristics about token lengths, finding them to be overly simplistic. We hope the insights of our study add clarity and intuition toward tokenization in contemporary LLMs. |
| title | How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.11518 |