Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2401.16818 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917639609122816 |
|---|---|
| author | Singer, Philipp Pfeiffer, Pascal Babakhin, Yauhen Jeblick, Maximilian Dhankhar, Nischay Fodor, Gabor Ambati, Sri Satish |
| author_facet | Singer, Philipp Pfeiffer, Pascal Babakhin, Yauhen Jeblick, Maximilian Dhankhar, Nischay Fodor, Gabor Ambati, Sri Satish |
| contents | We present H2O-Danube, a series of small 1.8B language models consisting of H2O-Danube-1.8B, trained on 1T tokens, and the incremental improved H2O-Danube2-1.8B trained on an additional 2T tokens. Our models exhibit highly competitive metrics across a multitude of benchmarks and, as of the time of this writing, H2O-Danube2-1.8B achieves the top ranking on Open LLM Leaderboard for all models below the 2B parameter range. The models follow core principles of LLama 2 and Mistral, and we leverage and refine various techniques for pre-training large language models. We additionally release chat models trained with supervised fine-tuning followed by direct preference optimization. We make all models openly available under Apache 2.0 license further democratizing LLMs to a wider audience economically. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_16818 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | H2O-Danube-1.8B Technical Report Singer, Philipp Pfeiffer, Pascal Babakhin, Yauhen Jeblick, Maximilian Dhankhar, Nischay Fodor, Gabor Ambati, Sri Satish Computation and Language Machine Learning We present H2O-Danube, a series of small 1.8B language models consisting of H2O-Danube-1.8B, trained on 1T tokens, and the incremental improved H2O-Danube2-1.8B trained on an additional 2T tokens. Our models exhibit highly competitive metrics across a multitude of benchmarks and, as of the time of this writing, H2O-Danube2-1.8B achieves the top ranking on Open LLM Leaderboard for all models below the 2B parameter range. The models follow core principles of LLama 2 and Mistral, and we leverage and refine various techniques for pre-training large language models. We additionally release chat models trained with supervised fine-tuning followed by direct preference optimization. We make all models openly available under Apache 2.0 license further democratizing LLMs to a wider audience economically. |
| title | H2O-Danube-1.8B Technical Report |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2401.16818 |