Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912966499106816 |
|---|---|
| author | Laiyk, Nurkhan Orel, Daniil Joshi, Rituraj Goloburda, Maiya Wang, Yuxia Nakov, Preslav Koto, Fajri |
| author_facet | Laiyk, Nurkhan Orel, Daniil Joshi, Rituraj Goloburda, Maiya Wang, Yuxia Nakov, Preslav Koto, Fajri |
| contents | Instruction tuning in low-resource languages remains underexplored due to limited text data, particularly in government and cultural domains. To address this, we introduce and open-source a large-scale (10,600 samples) instruction-following (IFT) dataset, covering key institutional and cultural knowledge relevant to Kazakhstan. Our dataset enhances LLMs' understanding of procedural, legal, and structural governance topics. We employ LLM-assisted data generation, comparing open-weight and closed-weight models for dataset construction, and select GPT-4o as the backbone. Each entity of our dataset undergoes full manual verification to ensure high quality. We also show that fine-tuning Qwen, Falcon, and Gemma on our dataset leads to consistent performance improvements in both multiple-choice and generative tasks, demonstrating the potential of LLM-assisted instruction tuning for low-resource languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13647 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh Laiyk, Nurkhan Orel, Daniil Joshi, Rituraj Goloburda, Maiya Wang, Yuxia Nakov, Preslav Koto, Fajri Computation and Language Instruction tuning in low-resource languages remains underexplored due to limited text data, particularly in government and cultural domains. To address this, we introduce and open-source a large-scale (10,600 samples) instruction-following (IFT) dataset, covering key institutional and cultural knowledge relevant to Kazakhstan. Our dataset enhances LLMs' understanding of procedural, legal, and structural governance topics. We employ LLM-assisted data generation, comparing open-weight and closed-weight models for dataset construction, and select GPT-4o as the backbone. Each entity of our dataset undergoes full manual verification to ensure high quality. We also show that fine-tuning Qwen, Falcon, and Gemma on our dataset leads to consistent performance improvements in both multiple-choice and generative tasks, demonstrating the potential of LLM-assisted instruction tuning for low-resource languages. |
| title | Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.13647 |