From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908410884128768 |
|---|---|
| author | Videau, Mathurin Idrissi, Badr Youbi Leite, Alessandro Schoenauer, Marc Teytaud, Olivier Lopez-Paz, David |
| author_facet | Videau, Mathurin Idrissi, Badr Youbi Leite, Alessandro Schoenauer, Marc Teytaud, Olivier Lopez-Paz, David |
| contents | Tokenization imposes a fixed granularity on the input text, freezing how a language model operates on data and how far in the future it predicts. Byte Pair Encoding (BPE) and similar schemes split text once, build a static vocabulary, and leave the model stuck with that choice. We relax this rigidity by introducing an autoregressive U-Net that learns to embed its own tokens as it trains. The network reads raw bytes, pools them into words, then pairs of words, then up to 4 words, giving it a multi-scale view of the sequence. At deeper stages, the model must predict further into the future -- anticipating the next few words rather than the next byte -- so deeper stages focus on broader semantic patterns while earlier stages handle fine details. When carefully tuning and controlling pretraining compute, shallow hierarchies tie strong BPE baselines, and deeper hierarchies have a promising trend. Because tokenization now lives inside the model, the same system can handle character-level tasks and carry knowledge across low-resource languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14761 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Bytes to Ideas: Language Modeling with Autoregressive U-Nets Videau, Mathurin Idrissi, Badr Youbi Leite, Alessandro Schoenauer, Marc Teytaud, Olivier Lopez-Paz, David Computation and Language Artificial Intelligence Tokenization imposes a fixed granularity on the input text, freezing how a language model operates on data and how far in the future it predicts. Byte Pair Encoding (BPE) and similar schemes split text once, build a static vocabulary, and leave the model stuck with that choice. We relax this rigidity by introducing an autoregressive U-Net that learns to embed its own tokens as it trains. The network reads raw bytes, pools them into words, then pairs of words, then up to 4 words, giving it a multi-scale view of the sequence. At deeper stages, the model must predict further into the future -- anticipating the next few words rather than the next byte -- so deeper stages focus on broader semantic patterns while earlier stages handle fine details. When carefully tuning and controlling pretraining compute, shallow hierarchies tie strong BPE baselines, and deeper hierarchies have a promising trend. Because tokenization now lives inside the model, the same system can handle character-level tasks and carry knowledge across low-resource languages. |
| title | From Bytes to Ideas: Language Modeling with Autoregressive U-Nets |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.14761 |