LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910571845124096 |
|---|---|
| author | Javed, Tahir Nawale, Janki Joshi, Sakshi George, Eldho Bhogale, Kaushal Mehendale, Deovrat Khapra, Mitesh M. |
| author_facet | Javed, Tahir Nawale, Janki Joshi, Sakshi George, Eldho Bhogale, Kaushal Mehendale, Deovrat Khapra, Mitesh M. |
| contents | Hindi, one of the most spoken language of India, exhibits a diverse array of accents due to its usage among individuals from diverse linguistic origins. To enable a robust evaluation of Hindi ASR systems on multiple accents, we create a benchmark, LAHAJA, which contains read and extempore speech on a diverse set of topics and use cases, with a total of 12.5 hours of Hindi audio, sourced from 132 speakers spanning 83 districts of India. We evaluate existing open-source and commercial models on LAHAJA and find their performance to be poor. We then train models using different datasets and find that our model trained on multilingual data with good speaker diversity outperforms existing models by a significant margin. We also present a fine-grained analysis which shows that the performance declines for speakers from North-East and South India, especially with content heavy in named entities and specialized terminology. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_11440 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems Javed, Tahir Nawale, Janki Joshi, Sakshi George, Eldho Bhogale, Kaushal Mehendale, Deovrat Khapra, Mitesh M. Computation and Language Hindi, one of the most spoken language of India, exhibits a diverse array of accents due to its usage among individuals from diverse linguistic origins. To enable a robust evaluation of Hindi ASR systems on multiple accents, we create a benchmark, LAHAJA, which contains read and extempore speech on a diverse set of topics and use cases, with a total of 12.5 hours of Hindi audio, sourced from 132 speakers spanning 83 districts of India. We evaluate existing open-source and commercial models on LAHAJA and find their performance to be poor. We then train models using different datasets and find that our model trained on multilingual data with good speaker diversity outperforms existing models by a significant margin. We also present a fine-grained analysis which shows that the performance declines for speakers from North-East and South India, especially with content heavy in named entities and specialized terminology. |
| title | LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2408.11440 |