WER We Stand: Benchmarking Urdu ASR Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909639822540800 |
|---|---|
| author | Arif, Samee Farid, Sualeha Khan, Aamina Jamal Abbas, Mustafa Raza, Agha Ali Athar, Awais |
| author_facet | Arif, Samee Farid, Sualeha Khan, Aamina Jamal Abbas, Mustafa Raza, Agha Ali Athar, Awais |
| contents | This paper presents a comprehensive evaluation of Urdu Automatic Speech Recognition (ASR) models. We analyze the performance of three ASR model families: Whisper, MMS, and Seamless-M4T using Word Error Rate (WER), along with a detailed examination of the most frequent wrong words and error types including insertions, deletions, and substitutions. Our analysis is conducted using two types of datasets, read speech and conversational speech. Notably, we present the first conversational speech dataset designed for benchmarking Urdu ASR models. We find that seamless-large outperforms other ASR models on the read speech dataset, while whisper-large performs best on the conversational speech dataset. Furthermore, this evaluation highlights the complexities of assessing ASR models for low-resource languages like Urdu using quantitative metrics alone and emphasizes the need for a robust Urdu text normalization system. Our findings contribute valuable insights for developing robust ASR systems for low-resource languages like Urdu. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_11252 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WER We Stand: Benchmarking Urdu ASR Models Arif, Samee Farid, Sualeha Khan, Aamina Jamal Abbas, Mustafa Raza, Agha Ali Athar, Awais Computation and Language Sound Audio and Speech Processing This paper presents a comprehensive evaluation of Urdu Automatic Speech Recognition (ASR) models. We analyze the performance of three ASR model families: Whisper, MMS, and Seamless-M4T using Word Error Rate (WER), along with a detailed examination of the most frequent wrong words and error types including insertions, deletions, and substitutions. Our analysis is conducted using two types of datasets, read speech and conversational speech. Notably, we present the first conversational speech dataset designed for benchmarking Urdu ASR models. We find that seamless-large outperforms other ASR models on the read speech dataset, while whisper-large performs best on the conversational speech dataset. Furthermore, this evaluation highlights the complexities of assessing ASR models for low-resource languages like Urdu using quantitative metrics alone and emphasizes the need for a robust Urdu text normalization system. Our findings contribute valuable insights for developing robust ASR systems for low-resource languages like Urdu. |
| title | WER We Stand: Benchmarking Urdu ASR Models |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.11252 |