Quantification of stylistic differences in human- and ASR-produced transcripts of African American English
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914937852395520 |
|---|---|
| author | Heuser, Annika Kendall, Tyler del Rio, Miguel McNamara, Quinten Bhandari, Nishchal Miller, Corey Jetté, Migüel |
| author_facet | Heuser, Annika Kendall, Tyler del Rio, Miguel McNamara, Quinten Bhandari, Nishchal Miller, Corey Jetté, Migüel |
| contents | Common measures of accuracy used to assess the performance of automatic speech recognition (ASR) systems, as well as human transcribers, conflate multiple sources of error. Stylistic differences, such as verbatim vs non-verbatim, can play a significant role in ASR performance evaluation when differences exist between training and test datasets. The problem is compounded for speech from underrepresented varieties, where the speech to orthography mapping is not as standardized. We categorize the kinds of stylistic differences between 6 transcription versions, 4 human- and 2 ASR-produced, of 10 hours of African American English (AAE) speech. Focusing on verbatim features and AAE morphosyntactic features, we investigate the interactions of these categories with how well transcripts can be compared via word error rate (WER). The results, and overall analysis, help clarify how ASR outputs are a function of the decisions made by the training data's human transcribers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_03059 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Quantification of stylistic differences in human- and ASR-produced transcripts of African American English Heuser, Annika Kendall, Tyler del Rio, Miguel McNamara, Quinten Bhandari, Nishchal Miller, Corey Jetté, Migüel Computation and Language Common measures of accuracy used to assess the performance of automatic speech recognition (ASR) systems, as well as human transcribers, conflate multiple sources of error. Stylistic differences, such as verbatim vs non-verbatim, can play a significant role in ASR performance evaluation when differences exist between training and test datasets. The problem is compounded for speech from underrepresented varieties, where the speech to orthography mapping is not as standardized. We categorize the kinds of stylistic differences between 6 transcription versions, 4 human- and 2 ASR-produced, of 10 hours of African American English (AAE) speech. Focusing on verbatim features and AAE morphosyntactic features, we investigate the interactions of these categories with how well transcripts can be compared via word error rate (WER). The results, and overall analysis, help clarify how ASR outputs are a function of the decisions made by the training data's human transcribers. |
| title | Quantification of stylistic differences in human- and ASR-produced transcripts of African American English |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.03059 |