ByteSpan: Information-Driven Subword Tokenisation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913907736576000 |
|---|---|
| author | Goriely, Zébulon Salhan, Suchir Lesci, Pietro Cheng, Julius Buttery, Paula |
| author_facet | Goriely, Zébulon Salhan, Suchir Lesci, Pietro Cheng, Julius Buttery, Paula |
| contents | Recent dynamic tokenisation methods operate directly on bytes and pool their latent representations into patches. This bears similarities to computational models of word segmentation that determine lexical boundaries using spikes in an autoregressive model's prediction error. Inspired by this connection, we explore whether grouping predictable bytes - rather than pooling their representations - can yield a useful fixed subword vocabulary. We propose a new information-driven subword tokeniser, ByteSpan, that uses an external byte-level LM during training to identify contiguous predictable byte sequences and group them into subwords. Experiments show that ByteSpan yields efficient vocabularies with higher morphological alignment scores than BPE for English. Multilingual experiments show similar compression and Rényi efficiency for 25 languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_18639 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ByteSpan: Information-Driven Subword Tokenisation Goriely, Zébulon Salhan, Suchir Lesci, Pietro Cheng, Julius Buttery, Paula Computation and Language Recent dynamic tokenisation methods operate directly on bytes and pool their latent representations into patches. This bears similarities to computational models of word segmentation that determine lexical boundaries using spikes in an autoregressive model's prediction error. Inspired by this connection, we explore whether grouping predictable bytes - rather than pooling their representations - can yield a useful fixed subword vocabulary. We propose a new information-driven subword tokeniser, ByteSpan, that uses an external byte-level LM during training to identify contiguous predictable byte sequences and group them into subwords. Experiments show that ByteSpan yields efficient vocabularies with higher morphological alignment scores than BPE for English. Multilingual experiments show similar compression and Rényi efficiency for 25 languages. |
| title | ByteSpan: Information-Driven Subword Tokenisation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.18639 |