Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910036815511552 |
|---|---|
| author | Bhogale, Kaushal Santosh Javed, Tahir John, Greeshma Susan Rathi, Dhruv Padmanaban, Akshayasree Parasa, Niharika Khapra, Mitesh M. |
| author_facet | Bhogale, Kaushal Santosh Javed, Tahir John, Greeshma Susan Rathi, Dhruv Padmanaban, Akshayasree Parasa, Niharika Khapra, Mitesh M. |
| contents | Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Word Error Rate (WER) often presents a bleaker picture of system performance than what human users perceive. Better aligning evaluation with real-world performance requires capturing permissible orthographic variations, which is extremely challenging for under-resourced Indian languages. Leveraging recent advances in LLMs, we propose a framework for creating benchmarks that capture permissible variations. Through extensive experiments, we demonstrate that OIWER, by accounting for orthographic variations, reduces pessimistic error rates (an average improvement of 6.3 points), narrows inflated model gaps (e.g., Gemini-Canary performance difference drops from 18.1 to 11.5 points), and aligns more closely with human perception than prior methods like WER-SN by 4.9 points. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_00941 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages Bhogale, Kaushal Santosh Javed, Tahir John, Greeshma Susan Rathi, Dhruv Padmanaban, Akshayasree Parasa, Niharika Khapra, Mitesh M. Computation and Language Sound Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Word Error Rate (WER) often presents a bleaker picture of system performance than what human users perceive. Better aligning evaluation with real-world performance requires capturing permissible orthographic variations, which is extremely challenging for under-resourced Indian languages. Leveraging recent advances in LLMs, we propose a framework for creating benchmarks that capture permissible variations. Through extensive experiments, we demonstrate that OIWER, by accounting for orthographic variations, reduces pessimistic error rates (an average improvement of 6.3 points), narrows inflated model gaps (e.g., Gemini-Canary performance difference drops from 18.1 to 11.5 points), and aligns more closely with human perception than prior methods like WER-SN by 4.9 points. |
| title | Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages |
| topic | Computation and Language Sound |
| url | https://arxiv.org/abs/2603.00941 |