Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916322537897984 |
|---|---|
| author | Parida, Shantipriya Panwar, Shakshi Lata, Kusum Mishra, Sanskruti Sekhar, Sambit |
| author_facet | Parida, Shantipriya Panwar, Shakshi Lata, Kusum Mishra, Sanskruti Sekhar, Sambit |
| contents | Large language models (LLMs) demonstrated transformative capabilities in many applications that require automatically generating responses based on human instruction. However, the major challenge for building LLMs, particularly in Indic languages, is the availability of high-quality data for building foundation LLMs. In this paper, we are proposing a large pre-train dataset in Hindi useful for the Indic language Hindi. We have collected the data span across several domains including major dialects in Hindi. The dataset contains 1.28 billion Hindi tokens. We have explained our pipeline including data collection, pre-processing, and availability for LLM pre-training. The proposed approach can be easily extended to other Indic and low-resource languages and will be available freely for LLM pre-training and LLM research purposes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_09855 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi Parida, Shantipriya Panwar, Shakshi Lata, Kusum Mishra, Sanskruti Sekhar, Sambit Computation and Language Artificial Intelligence Large language models (LLMs) demonstrated transformative capabilities in many applications that require automatically generating responses based on human instruction. However, the major challenge for building LLMs, particularly in Indic languages, is the availability of high-quality data for building foundation LLMs. In this paper, we are proposing a large pre-train dataset in Hindi useful for the Indic language Hindi. We have collected the data span across several domains including major dialects in Hindi. The dataset contains 1.28 billion Hindi tokens. We have explained our pipeline including data collection, pre-processing, and availability for LLM pre-training. The proposed approach can be easily extended to other Indic and low-resource languages and will be available freely for LLM pre-training and LLM research purposes. |
| title | Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.09855 |