Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.15889 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916800370835456 |
|---|---|
| author | Hu, Yifan Liang, Frank Zhao, Dachuan Geuter, Jonathan Reddy, Varshini Schmidt, Craig W. Tanner, Chris |
| author_facet | Hu, Yifan Liang, Frank Zhao, Dachuan Geuter, Jonathan Reddy, Varshini Schmidt, Craig W. Tanner, Chris |
| contents | Byte-Pair Encoding (BPE) has become a widely adopted subword tokenization method in modern language models due to its simplicity and strong empirical performance across downstream tasks. However, applying BPE to unsegmented languages such as Chinese presents significant challenges, as its frequency-driven merge operation is agnostic to linguistic boundaries. To address this, we propose two entropy-informed pre-tokenization strategies that guide BPE segmentation using unsupervised information-theoretic cues. The first approach uses pointwise mutual information and left/right entropy to identify coherent character spans, while the second leverages predictive entropy derived from a pretrained GPT-2 model to detect boundary uncertainty. We evaluate both methods on a subset of the PKU dataset and demonstrate substantial improvements in segmentation precision, recall, and F1 score compared to standard BPE. Our results suggest that entropy-guided pre-tokenization not only enhances alignment with gold-standard linguistic units but also offers a promising direction for improving tokenization quality in low-resource and multilingual settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_15889 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Entropy-Driven Pre-Tokenization for Byte-Pair Encoding Hu, Yifan Liang, Frank Zhao, Dachuan Geuter, Jonathan Reddy, Varshini Schmidt, Craig W. Tanner, Chris Computation and Language Byte-Pair Encoding (BPE) has become a widely adopted subword tokenization method in modern language models due to its simplicity and strong empirical performance across downstream tasks. However, applying BPE to unsegmented languages such as Chinese presents significant challenges, as its frequency-driven merge operation is agnostic to linguistic boundaries. To address this, we propose two entropy-informed pre-tokenization strategies that guide BPE segmentation using unsupervised information-theoretic cues. The first approach uses pointwise mutual information and left/right entropy to identify coherent character spans, while the second leverages predictive entropy derived from a pretrained GPT-2 model to detect boundary uncertainty. We evaluate both methods on a subset of the PKU dataset and demonstrate substantial improvements in segmentation precision, recall, and F1 score compared to standard BPE. Our results suggest that entropy-guided pre-tokenization not only enhances alignment with gold-standard linguistic units but also offers a promising direction for improving tokenization quality in low-resource and multilingual settings. |
| title | Entropy-Driven Pre-Tokenization for Byte-Pair Encoding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.15889 |