Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.05668 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909774305558528 |
|---|---|
| author | Hoffmann, Michael John, Jophin Schweter, Stefan Ramakrishnan, Gokul Mak, Hoi-Fong Zhang, Alice Gaynullin, Dmitry Hammer, Nicolay J. |
| author_facet | Hoffmann, Michael John, Jophin Schweter, Stefan Ramakrishnan, Gokul Mak, Hoi-Fong Zhang, Alice Gaynullin, Dmitry Hammer, Nicolay J. |
| contents | We present Llama-GENBA-10B, a trilingual foundation model addressing English-centric bias in large language models. Built on Llama 3.1-8B and scaled to 10B parameters, Llama-GENBA-10B is continuously pretrained on 164B tokens (82B English, 82B German, and 80M Bavarian), balancing resources while preventing English dominance. Targeted at the German NLP community, the model also promotes Bavarian as a low-resource language. Development tackled four challenges: (1) curating a multilingual corpus despite Bavarian scarcity, (2) creating a unified tokenizer for English, German, and Bavarian, (3) optimizing architecture and language-ratio hyperparameters for cross-lingual transfer, and (4) establishing the first standardized trilingual evaluation suite by translating German benchmarks into Bavarian. Evaluations show that Llama-GENBA-10B achieves strong cross-lingual performance, with the fine-tuned variant surpassing Apertus-8B-2509 and gemma-2-9b in Bavarian and establishing itself as the best model in its class for this language, while also outperforming EuroLLM in English and matching its results in German. Training on the Cerebras CS-2 demonstrated efficient large-scale multilingual pretraining with documented energy use, offering a blueprint for inclusive foundation models that integrate low-resource languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_05668 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian Hoffmann, Michael John, Jophin Schweter, Stefan Ramakrishnan, Gokul Mak, Hoi-Fong Zhang, Alice Gaynullin, Dmitry Hammer, Nicolay J. Computation and Language Artificial Intelligence We present Llama-GENBA-10B, a trilingual foundation model addressing English-centric bias in large language models. Built on Llama 3.1-8B and scaled to 10B parameters, Llama-GENBA-10B is continuously pretrained on 164B tokens (82B English, 82B German, and 80M Bavarian), balancing resources while preventing English dominance. Targeted at the German NLP community, the model also promotes Bavarian as a low-resource language. Development tackled four challenges: (1) curating a multilingual corpus despite Bavarian scarcity, (2) creating a unified tokenizer for English, German, and Bavarian, (3) optimizing architecture and language-ratio hyperparameters for cross-lingual transfer, and (4) establishing the first standardized trilingual evaluation suite by translating German benchmarks into Bavarian. Evaluations show that Llama-GENBA-10B achieves strong cross-lingual performance, with the fine-tuned variant surpassing Apertus-8B-2509 and gemma-2-9b in Bavarian and establishing itself as the best model in its class for this language, while also outperforming EuroLLM in English and matching its results in German. Training on the Cerebras CS-2 demonstrated efficient large-scale multilingual pretraining with documented energy use, offering a blueprint for inclusive foundation models that integrate low-resource languages. |
| title | Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.05668 |