Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.15953 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912969411002368 |
|---|---|
| author | Alpha, Aleph : Abdessaied, Adnen Baranowski, Artur Balles, Lukas Barlow, Michael Benureau, Fabien C. Y. Berkenkamp, Felix Bluebaum, Lukas Boll, Bastian Burns, Thomas F. Deiseroth, Björn Eichenberg, Constantin Friede, David Guerrero, Pablo Iyu Hammam, Ahmed Harren, Bastian Higl, Johann Jadidi, Yasser Kauf, Carina Messner, Johannes Metzen, Jan Hendrik Meuer, Max Nanda, Vedant Neitemeier, Pit Oostermeijer, Koen Parcalabescu, Letitia Pernpointner, Markus Reinfurt, Felix Rodriquez, Dylan Schott, Grégory Siedler, Philipp Simonovsky, Martin Speicher, Till Stampa, Volker Wäldchen, Stephan Weinbach, Samuel Ziegltrum, Gregor |
| author_facet | Alpha, Aleph : Abdessaied, Adnen Baranowski, Artur Balles, Lukas Barlow, Michael Benureau, Fabien C. Y. Berkenkamp, Felix Bluebaum, Lukas Boll, Bastian Burns, Thomas F. Deiseroth, Björn Eichenberg, Constantin Friede, David Guerrero, Pablo Iyu Hammam, Ahmed Harren, Bastian Higl, Johann Jadidi, Yasser Kauf, Carina Messner, Johannes Metzen, Jan Hendrik Meuer, Max Nanda, Vedant Neitemeier, Pit Oostermeijer, Koen Parcalabescu, Letitia Pernpointner, Markus Reinfurt, Felix Rodriquez, Dylan Schott, Grégory Siedler, Philipp Simonovsky, Martin Speicher, Till Stampa, Volker Wäldchen, Stephan Weinbach, Samuel Ziegltrum, Gregor |
| contents | Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although learned tokenizers are widely adopted, they exhibit notable limitations, including their large, fixed vocabulary sizes and poor adaptability to new domains or languages. We present a family of models with up to 70 billion parameters based on the hierarchical autoregressive transformer (HAT) architecture. In HAT, an encoder transformer aggregates bytes into word embeddings and then feeds them to the backbone, a classical autoregressive transformer. The outputs of the backbone are then cross-attended by the decoder and converted back into bytes. We show that we can reuse available pre-trained models by converting the Llama 3.1 8B and 70B models into the HAT architecture: Llama-3.1-8B-TFree-HAT and Llama-3.1-70B-TFree-HAT are byte-level models whose encoder and decoder are trained from scratch, but where we adapt the pre-trained Llama backbone, i.e., the transformer blocks with the embedding matrix and head removed, to handle word embeddings instead of the original tokens. We also provide a 7B HAT model, Llama-TFree-HAT-Pretrained, trained entirely from scratch on nearly 4 trillion words. The HAT architecture improves text compression by reducing the number of required sequence positions and enhances robustness to intra-word variations, e.g., spelling differences. Through pre-training, as well as subsequent supervised fine-tuning and direct preference optimization in English and German, we show strong proficiency in both languages, improving on the original Llama 3.1 in most benchmarks. We release our models (including 200 pre-training checkpoints) on Hugging Face. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15953 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Family of LLMs Liberated from Static Vocabularies Alpha, Aleph : Abdessaied, Adnen Baranowski, Artur Balles, Lukas Barlow, Michael Benureau, Fabien C. Y. Berkenkamp, Felix Bluebaum, Lukas Boll, Bastian Burns, Thomas F. Deiseroth, Björn Eichenberg, Constantin Friede, David Guerrero, Pablo Iyu Hammam, Ahmed Harren, Bastian Higl, Johann Jadidi, Yasser Kauf, Carina Messner, Johannes Metzen, Jan Hendrik Meuer, Max Nanda, Vedant Neitemeier, Pit Oostermeijer, Koen Parcalabescu, Letitia Pernpointner, Markus Reinfurt, Felix Rodriquez, Dylan Schott, Grégory Siedler, Philipp Simonovsky, Martin Speicher, Till Stampa, Volker Wäldchen, Stephan Weinbach, Samuel Ziegltrum, Gregor Computation and Language Artificial Intelligence Machine Learning Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although learned tokenizers are widely adopted, they exhibit notable limitations, including their large, fixed vocabulary sizes and poor adaptability to new domains or languages. We present a family of models with up to 70 billion parameters based on the hierarchical autoregressive transformer (HAT) architecture. In HAT, an encoder transformer aggregates bytes into word embeddings and then feeds them to the backbone, a classical autoregressive transformer. The outputs of the backbone are then cross-attended by the decoder and converted back into bytes. We show that we can reuse available pre-trained models by converting the Llama 3.1 8B and 70B models into the HAT architecture: Llama-3.1-8B-TFree-HAT and Llama-3.1-70B-TFree-HAT are byte-level models whose encoder and decoder are trained from scratch, but where we adapt the pre-trained Llama backbone, i.e., the transformer blocks with the embedding matrix and head removed, to handle word embeddings instead of the original tokens. We also provide a 7B HAT model, Llama-TFree-HAT-Pretrained, trained entirely from scratch on nearly 4 trillion words. The HAT architecture improves text compression by reducing the number of required sequence positions and enhances robustness to intra-word variations, e.g., spelling differences. Through pre-training, as well as subsequent supervised fine-tuning and direct preference optimization in English and German, we show strong proficiency in both languages, improving on the original Llama 3.1 in most benchmarks. We release our models (including 200 pre-training checkpoints) on Hugging Face. |
| title | A Family of LLMs Liberated from Static Vocabularies |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2603.15953 |