What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929585233330176 |
|---|---|
| author | Manohar, Kavya Pillai, Leena G Sherly, Elizabeth |
| author_facet | Manohar, Kavya Pillai, Leena G Sherly, Elizabeth |
| contents | This paper explores the pitfalls in evaluating multilingual automatic speech recognition (ASR) models, with a particular focus on Indic language scripts. We investigate the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta's MMS, Seamless, and Assembly AI's Conformer, and their unintended consequences on performance metrics. Our research reveals that current text normalization practices, while aiming to standardize ASR outputs for fair comparison, by removing inconsistencies such as variations in spelling, punctuation, and special characters, are fundamentally flawed when applied to Indic scripts. Through empirical analysis using text similarity scores and in-depth linguistic examination, we demonstrate that these flaws lead to artificially improved performance metrics for Indic languages. We conclude by proposing a shift towards developing text normalization routines that leverage native linguistic expertise, ensuring more robust and accurate evaluations of multilingual ASR models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_02449 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations Manohar, Kavya Pillai, Leena G Sherly, Elizabeth Computation and Language Artificial Intelligence Human-Computer Interaction 68T50, 91F20, 68T10 I.2.1; I.2.7 This paper explores the pitfalls in evaluating multilingual automatic speech recognition (ASR) models, with a particular focus on Indic language scripts. We investigate the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta's MMS, Seamless, and Assembly AI's Conformer, and their unintended consequences on performance metrics. Our research reveals that current text normalization practices, while aiming to standardize ASR outputs for fair comparison, by removing inconsistencies such as variations in spelling, punctuation, and special characters, are fundamentally flawed when applied to Indic scripts. Through empirical analysis using text similarity scores and in-depth linguistic examination, we demonstrate that these flaws lead to artificially improved performance metrics for Indic languages. We conclude by proposing a shift towards developing text normalization routines that leverage native linguistic expertise, ensuring more robust and accurate evaluations of multilingual ASR models. |
| title | What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations |
| topic | Computation and Language Artificial Intelligence Human-Computer Interaction 68T50, 91F20, 68T10 I.2.1; I.2.7 |
| url | https://arxiv.org/abs/2409.02449 |