Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909831891255296 |
|---|---|
| author | Koto, Fajri Joshi, Rituraj Mukhituly, Nurdaulet Wang, Yuxia Xie, Zhuohan Pal, Rahul Orel, Daniil Mullah, Parvez Turmakhan, Diana Goloburda, Maiya Kamran, Mohammed Ghosh, Samujjwal Jia, Bokang Mansurov, Jonibek Togmanov, Mukhammed Banerjee, Debopriyo Laiyk, Nurkhan Sakip, Akhmed Han, Xudong Kochmar, Ekaterina Aji, Alham Fikri Singh, Aaryamonvikram Jadhav, Alok Anil Katipomu, Satheesh Kamboj, Samta Choudhury, Monojit Gosal, Gurpreet Ramakrishnan, Gokulakrishnan Mishra, Biswajit Chandran, Sarath Sheinin, Avraham Vassilieva, Natalia Sengupta, Neha Nakov, Preslav |
| author_facet | Koto, Fajri Joshi, Rituraj Mukhituly, Nurdaulet Wang, Yuxia Xie, Zhuohan Pal, Rahul Orel, Daniil Mullah, Parvez Turmakhan, Diana Goloburda, Maiya Kamran, Mohammed Ghosh, Samujjwal Jia, Bokang Mansurov, Jonibek Togmanov, Mukhammed Banerjee, Debopriyo Laiyk, Nurkhan Sakip, Akhmed Han, Xudong Kochmar, Ekaterina Aji, Alham Fikri Singh, Aaryamonvikram Jadhav, Alok Anil Katipomu, Satheesh Kamboj, Samta Choudhury, Monojit Gosal, Gurpreet Ramakrishnan, Gokulakrishnan Mishra, Biswajit Chandran, Sarath Sheinin, Avraham Vassilieva, Natalia Sengupta, Neha Nakov, Preslav |
| contents | Llama-3.1-Sherkala-8B-Chat, or Sherkala-Chat (8B) for short, is a state-of-the-art instruction-tuned open generative large language model (LLM) designed for Kazakh. Sherkala-Chat (8B) aims to enhance the inclusivity of LLM advancements for Kazakh speakers. Adapted from the LLaMA-3.1-8B model, Sherkala-Chat (8B) is trained on 45.3B tokens across Kazakh, English, Russian, and Turkish. With 8 billion parameters, it demonstrates strong knowledge and reasoning abilities in Kazakh, significantly outper-forming existing open Kazakh and multilingual models of similar scale while achieving competitive performance in English. To ensure effective and responsible alignment, we leverage translated instruction datasets, a Kazakhstan-specific instruction dataset that is automatically constructed and manually verified, and Kazakh-specific safety data. We release Sherkala-Chat (8B) as an open-weight model, along with a detailed description of its training, alignment, and evaluation, to support research and real-world applications for Kazakh speakers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01493 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting Koto, Fajri Joshi, Rituraj Mukhituly, Nurdaulet Wang, Yuxia Xie, Zhuohan Pal, Rahul Orel, Daniil Mullah, Parvez Turmakhan, Diana Goloburda, Maiya Kamran, Mohammed Ghosh, Samujjwal Jia, Bokang Mansurov, Jonibek Togmanov, Mukhammed Banerjee, Debopriyo Laiyk, Nurkhan Sakip, Akhmed Han, Xudong Kochmar, Ekaterina Aji, Alham Fikri Singh, Aaryamonvikram Jadhav, Alok Anil Katipomu, Satheesh Kamboj, Samta Choudhury, Monojit Gosal, Gurpreet Ramakrishnan, Gokulakrishnan Mishra, Biswajit Chandran, Sarath Sheinin, Avraham Vassilieva, Natalia Sengupta, Neha Nakov, Preslav Computation and Language Llama-3.1-Sherkala-8B-Chat, or Sherkala-Chat (8B) for short, is a state-of-the-art instruction-tuned open generative large language model (LLM) designed for Kazakh. Sherkala-Chat (8B) aims to enhance the inclusivity of LLM advancements for Kazakh speakers. Adapted from the LLaMA-3.1-8B model, Sherkala-Chat (8B) is trained on 45.3B tokens across Kazakh, English, Russian, and Turkish. With 8 billion parameters, it demonstrates strong knowledge and reasoning abilities in Kazakh, significantly outper-forming existing open Kazakh and multilingual models of similar scale while achieving competitive performance in English. To ensure effective and responsible alignment, we leverage translated instruction datasets, a Kazakhstan-specific instruction dataset that is automatically constructed and manually verified, and Kazakh-specific safety data. We release Sherkala-Chat (8B) as an open-weight model, along with a detailed description of its training, alignment, and evaluation, to support research and real-world applications for Kazakh speakers. |
| title | Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.01493 |