Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929274362003456 |
|---|---|
| author | Shor, Joel Bi, Ruyue Agnes Venugopalan, Subhashini Ibara, Steven Goldenberg, Roman Rivlin, Ehud |
| author_facet | Shor, Joel Bi, Ruyue Agnes Venugopalan, Subhashini Ibara, Steven Goldenberg, Roman Rivlin, Ehud |
| contents | Automatic Speech Recognition (ASR) in medical contexts has the potential to save time, cut costs, increase report accuracy, and reduce physician burnout. However, the healthcare industry has been slower to adopt this technology, in part due to the importance of avoiding medically-relevant transcription mistakes. In this work, we present the Clinical BERTScore (CBERTScore), an ASR metric that penalizes clinically-relevant mistakes more than others. We demonstrate that this metric more closely aligns with clinician preferences on medical sentences as compared to other metrics (WER, BLUE, METEOR, etc), sometimes by wide margins. We collect a benchmark of 18 clinician preferences on 149 realistic medical sentences called the Clinician Transcript Preference benchmark (CTP) and make it publicly available for the community to further develop clinically-aware ASR metrics. To our knowledge, this is the first public dataset of its kind. We demonstrate that CBERTScore more closely matches what clinicians prefer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2303_05737 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings Shor, Joel Bi, Ruyue Agnes Venugopalan, Subhashini Ibara, Steven Goldenberg, Roman Rivlin, Ehud Audio and Speech Processing Computation and Language Machine Learning Sound Automatic Speech Recognition (ASR) in medical contexts has the potential to save time, cut costs, increase report accuracy, and reduce physician burnout. However, the healthcare industry has been slower to adopt this technology, in part due to the importance of avoiding medically-relevant transcription mistakes. In this work, we present the Clinical BERTScore (CBERTScore), an ASR metric that penalizes clinically-relevant mistakes more than others. We demonstrate that this metric more closely aligns with clinician preferences on medical sentences as compared to other metrics (WER, BLUE, METEOR, etc), sometimes by wide margins. We collect a benchmark of 18 clinician preferences on 149 realistic medical sentences called the Clinician Transcript Preference benchmark (CTP) and make it publicly available for the community to further develop clinically-aware ASR metrics. To our knowledge, this is the first public dataset of its kind. We demonstrate that CBERTScore more closely matches what clinicians prefer. |
| title | Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings |
| topic | Audio and Speech Processing Computation and Language Machine Learning Sound |
| url | https://arxiv.org/abs/2303.05737 |