CorIL: Towards Enriching Indian Language to Indian Language Parallel Corpora and Machine Translation Systems
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912602546765824 |
|---|---|
| author | Bhattacharjee, Soham Roy, Mukund K Poojary, Yathish Dave, Bhargav Raj, Mihir Mujadia, Vandan Gain, Baban Mishra, Pruthwik Ahsan, Arafat Krishnamurthy, Parameswari Rao, Ashwath Josan, Gurpreet Singh Dubey, Preeti Kak, Aadil Amin Kulkarni, Anna Rao VG, Narendra Arora, Sunita Balbantray, Rakesh Majumdar, Prasenjit Arora, Karunesh K Ekbal, Asif Sharma, Dipti Mishra |
| author_facet | Bhattacharjee, Soham Roy, Mukund K Poojary, Yathish Dave, Bhargav Raj, Mihir Mujadia, Vandan Gain, Baban Mishra, Pruthwik Ahsan, Arafat Krishnamurthy, Parameswari Rao, Ashwath Josan, Gurpreet Singh Dubey, Preeti Kak, Aadil Amin Kulkarni, Anna Rao VG, Narendra Arora, Sunita Balbantray, Rakesh Majumdar, Prasenjit Arora, Karunesh K Ekbal, Asif Sharma, Dipti Mishra |
| contents | India's linguistic landscape is one of the most diverse in the world, comprising over 120 major languages and approximately 1,600 additional languages, with 22 officially recognized as scheduled languages in the Indian Constitution. Despite recent progress in multilingual neural machine translation (NMT), high-quality parallel corpora for Indian languages remain scarce, especially across varied domains. In this paper, we introduce a large-scale, high-quality annotated parallel corpus covering 11 of these languages : English, Telugu, Hindi, Punjabi, Odia, Kashmiri, Sindhi, Dogri, Kannada, Urdu, and Gujarati comprising a total of 772,000 bi-text sentence pairs. The dataset is carefully curated and systematically categorized into three key domains: Government, Health, and General, to enable domain-aware machine translation research and facilitate effective domain adaptation. To demonstrate the utility of CorIL and establish strong benchmarks for future research, we fine-tune and evaluate several state-of-the-art NMT models, including IndicTrans2, NLLB, and BhashaVerse. Our analysis reveals important performance trends and highlights the corpus's value in probing model capabilities. For instance, the results show distinct performance patterns based on language script, with massively multilingual models showing an advantage on Perso-Arabic scripts (Urdu, Sindhi) while other models excel on Indic scripts. This paper provides a detailed domain-wise performance analysis, offering insights into domain sensitivity and cross-script transfer learning. By publicly releasing CorIL, we aim to significantly improve the availability of high-quality training data for Indian languages and provide a valuable resource for the machine translation research community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19941 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CorIL: Towards Enriching Indian Language to Indian Language Parallel Corpora and Machine Translation Systems Bhattacharjee, Soham Roy, Mukund K Poojary, Yathish Dave, Bhargav Raj, Mihir Mujadia, Vandan Gain, Baban Mishra, Pruthwik Ahsan, Arafat Krishnamurthy, Parameswari Rao, Ashwath Josan, Gurpreet Singh Dubey, Preeti Kak, Aadil Amin Kulkarni, Anna Rao VG, Narendra Arora, Sunita Balbantray, Rakesh Majumdar, Prasenjit Arora, Karunesh K Ekbal, Asif Sharma, Dipti Mishra Computation and Language Artificial Intelligence India's linguistic landscape is one of the most diverse in the world, comprising over 120 major languages and approximately 1,600 additional languages, with 22 officially recognized as scheduled languages in the Indian Constitution. Despite recent progress in multilingual neural machine translation (NMT), high-quality parallel corpora for Indian languages remain scarce, especially across varied domains. In this paper, we introduce a large-scale, high-quality annotated parallel corpus covering 11 of these languages : English, Telugu, Hindi, Punjabi, Odia, Kashmiri, Sindhi, Dogri, Kannada, Urdu, and Gujarati comprising a total of 772,000 bi-text sentence pairs. The dataset is carefully curated and systematically categorized into three key domains: Government, Health, and General, to enable domain-aware machine translation research and facilitate effective domain adaptation. To demonstrate the utility of CorIL and establish strong benchmarks for future research, we fine-tune and evaluate several state-of-the-art NMT models, including IndicTrans2, NLLB, and BhashaVerse. Our analysis reveals important performance trends and highlights the corpus's value in probing model capabilities. For instance, the results show distinct performance patterns based on language script, with massively multilingual models showing an advantage on Perso-Arabic scripts (Urdu, Sindhi) while other models excel on Indic scripts. This paper provides a detailed domain-wise performance analysis, offering insights into domain sensitivity and cross-script transfer learning. By publicly releasing CorIL, we aim to significantly improve the availability of high-quality training data for Indian languages and provide a valuable resource for the machine translation research community. |
| title | CorIL: Towards Enriching Indian Language to Indian Language Parallel Corpora and Machine Translation Systems |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.19941 |