TinyLlama: An Open-Source Small Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910470167855104 |
|---|---|
| author | Zhang, Peiyuan Zeng, Guangtao Wang, Tianduo Lu, Wei |
| author_facet | Zhang, Peiyuan Zeng, Guangtao Wang, Tianduo Lu, Wei |
| contents | We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of Llama 2, TinyLlama leverages various advances contributed by the open-source community (e.g., FlashAttention and Lit-GPT), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes. Our model checkpoints and code are publicly available on GitHub at https://github.com/jzhang38/TinyLlama. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_02385 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TinyLlama: An Open-Source Small Language Model Zhang, Peiyuan Zeng, Guangtao Wang, Tianduo Lu, Wei Computation and Language Artificial Intelligence We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of Llama 2, TinyLlama leverages various advances contributed by the open-source community (e.g., FlashAttention and Lit-GPT), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes. Our model checkpoints and code are publicly available on GitHub at https://github.com/jzhang38/TinyLlama. |
| title | TinyLlama: An Open-Source Small Language Model |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2401.02385 |