BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911267715809280 |
|---|---|
| author | Manoj, Guduru Rachamalla, Neel Prabhanjan Kulkarni, Ashish Rajeev, Gautam Piplodiya, Jay Menezes, Arul Khan, Shaharukh Rana, Souvik Sah, Manya Khatri, Chandra Agarwal, Shubham |
| author_facet | Manoj, Guduru Rachamalla, Neel Prabhanjan Kulkarni, Ashish Rajeev, Gautam Piplodiya, Jay Menezes, Arul Khan, Shaharukh Rana, Souvik Sah, Manya Khatri, Chandra Agarwal, Shubham |
| contents | In the context of pretraining of Large Language Models (LLMs), synthetic data has emerged as an alternative for generating high-quality pretraining data at scale. This is particularly beneficial in low-resource language settings where the benefits of recent LLMs have been unevenly distributed across languages. In this work, we present a systematic study on the generation and evaluation of synthetic multilingual pretraining data for Indic languages, where we construct a large-scale synthetic dataset BhashaKritika, comprising 540B tokens using 5 different techniques for 10 languages. We explore the impact of grounding generation in documents, personas, and topics. We analyze how language choice, both in the prompt instructions and document grounding, affects data quality, and we compare translations of English content with native generation in Indic languages. To support scalable and language-sensitive evaluation, we introduce a modular quality evaluation pipeline that integrates script and language detection, metadata consistency checks, n-gram repetition analysis, and perplexity-based filtering using KenLM models. Our framework enables robust quality control across diverse scripts and linguistic contexts. Empirical results through model runs reveal key trade-offs in generation strategies and highlight best practices for constructing effective multilingual corpora. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_10338 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages Manoj, Guduru Rachamalla, Neel Prabhanjan Kulkarni, Ashish Rajeev, Gautam Piplodiya, Jay Menezes, Arul Khan, Shaharukh Rana, Souvik Sah, Manya Khatri, Chandra Agarwal, Shubham Computation and Language Artificial Intelligence In the context of pretraining of Large Language Models (LLMs), synthetic data has emerged as an alternative for generating high-quality pretraining data at scale. This is particularly beneficial in low-resource language settings where the benefits of recent LLMs have been unevenly distributed across languages. In this work, we present a systematic study on the generation and evaluation of synthetic multilingual pretraining data for Indic languages, where we construct a large-scale synthetic dataset BhashaKritika, comprising 540B tokens using 5 different techniques for 10 languages. We explore the impact of grounding generation in documents, personas, and topics. We analyze how language choice, both in the prompt instructions and document grounding, affects data quality, and we compare translations of English content with native generation in Indic languages. To support scalable and language-sensitive evaluation, we introduce a modular quality evaluation pipeline that integrates script and language detection, metadata consistency checks, n-gram repetition analysis, and perplexity-based filtering using KenLM models. Our framework enables robust quality control across diverse scripts and linguistic contexts. Empirical results through model runs reveal key trade-offs in generation strategies and highlight best practices for constructing effective multilingual corpora. |
| title | BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2511.10338 |