PhoGPT: Generative Pre-training for Vietnamese
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913277478436864 |
|---|---|
| author | Nguyen, Dat Quoc Nguyen, Linh The Tran, Chi Nguyen, Dung Ngoc Phung, Dinh Bui, Hung |
| author_facet | Nguyen, Dat Quoc Nguyen, Linh The Tran, Chi Nguyen, Dung Ngoc Phung, Dinh Bui, Hung |
| contents | We open-source a state-of-the-art 4B-parameter generative model series for Vietnamese, which includes the base pre-trained monolingual model PhoGPT-4B and its chat variant, PhoGPT-4B-Chat. The base model, PhoGPT-4B, with exactly 3.7B parameters, is pre-trained from scratch on a Vietnamese corpus of 102B tokens, with an 8192 context length, employing a vocabulary of 20480 token types. The chat variant, PhoGPT-4B-Chat, is the modeling output obtained by fine-tuning PhoGPT-4B on a dataset of 70K instructional prompts and their responses, along with an additional 290K conversations. In addition, we also demonstrate its superior performance compared to previous open-source models. Our PhoGPT models are available at: https://github.com/VinAIResearch/PhoGPT |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_02945 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | PhoGPT: Generative Pre-training for Vietnamese Nguyen, Dat Quoc Nguyen, Linh The Tran, Chi Nguyen, Dung Ngoc Phung, Dinh Bui, Hung Computation and Language We open-source a state-of-the-art 4B-parameter generative model series for Vietnamese, which includes the base pre-trained monolingual model PhoGPT-4B and its chat variant, PhoGPT-4B-Chat. The base model, PhoGPT-4B, with exactly 3.7B parameters, is pre-trained from scratch on a Vietnamese corpus of 102B tokens, with an 8192 context length, employing a vocabulary of 20480 token types. The chat variant, PhoGPT-4B-Chat, is the modeling output obtained by fine-tuning PhoGPT-4B on a dataset of 70K instructional prompts and their responses, along with an additional 290K conversations. In addition, we also demonstrate its superior performance compared to previous open-source models. Our PhoGPT models are available at: https://github.com/VinAIResearch/PhoGPT |
| title | PhoGPT: Generative Pre-training for Vietnamese |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.02945 |