Style-agnostic evaluation of ASR using multiple reference transcripts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917865859317760 |
|---|---|
| author | McNamara, Quinten Fernández, Miguel Ángel del Río Bhandari, Nishchal Ratajczak, Martin Chen, Danny Miller, Corey Jetté, Migüel |
| author_facet | McNamara, Quinten Fernández, Miguel Ángel del Río Bhandari, Nishchal Ratajczak, Martin Chen, Danny Miller, Corey Jetté, Migüel |
| contents | Word error rate (WER) as a metric has a variety of limitations that have plagued the field of speech recognition. Evaluation datasets suffer from varying style, formality, and inherent ambiguity of the transcription task. In this work, we attempt to mitigate some of these differences by performing style-agnostic evaluation of ASR systems using multiple references transcribed under opposing style parameters. As a result, we find that existing WER reports are likely significantly over-estimating the number of contentful errors made by state-of-the-art ASR systems. In addition, we have found our multireference method to be a useful mechanism for comparing the quality of ASR models that differ in the stylistic makeup of their training data and target task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07937 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Style-agnostic evaluation of ASR using multiple reference transcripts McNamara, Quinten Fernández, Miguel Ángel del Río Bhandari, Nishchal Ratajczak, Martin Chen, Danny Miller, Corey Jetté, Migüel Computation and Language Word error rate (WER) as a metric has a variety of limitations that have plagued the field of speech recognition. Evaluation datasets suffer from varying style, formality, and inherent ambiguity of the transcription task. In this work, we attempt to mitigate some of these differences by performing style-agnostic evaluation of ASR systems using multiple references transcribed under opposing style parameters. As a result, we find that existing WER reports are likely significantly over-estimating the number of contentful errors made by state-of-the-art ASR systems. In addition, we have found our multireference method to be a useful mechanism for comparing the quality of ASR models that differ in the stylistic makeup of their training data and target task. |
| title | Style-agnostic evaluation of ASR using multiple reference transcripts |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.07937 |