Bit-level BPE: Below the byte boundary
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908399547973632 |
|---|---|
| author | Moon, Sangwhan Hiraoka, Tatsuya Okazaki, Naoaki |
| author_facet | Moon, Sangwhan Hiraoka, Tatsuya Okazaki, Naoaki |
| contents | Byte-level fallbacks for subword tokenization have become a common practice in large language models. In particular, it has been demonstrated to be incredibly effective as a pragmatic solution for preventing OOV, especially in the context of larger models. However, breaking a character down to individual bytes significantly increases the sequence length for long-tail tokens in languages such as Chinese, Japanese, and Korean (CJK) and other character-diverse contexts such as emoji. The increased sequence length results in longer computation during both training and inference. In this work, we propose a simple compression technique that reduces the sequence length losslessly. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_07541 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Bit-level BPE: Below the byte boundary Moon, Sangwhan Hiraoka, Tatsuya Okazaki, Naoaki Computation and Language Byte-level fallbacks for subword tokenization have become a common practice in large language models. In particular, it has been demonstrated to be incredibly effective as a pragmatic solution for preventing OOV, especially in the context of larger models. However, breaking a character down to individual bytes significantly increases the sequence length for long-tail tokens in languages such as Chinese, Japanese, and Korean (CJK) and other character-diverse contexts such as emoji. The increased sequence length results in longer computation during both training and inference. In this work, we propose a simple compression technique that reduces the sequence length losslessly. |
| title | Bit-level BPE: Below the byte boundary |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.07541 |