When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914528181092352 |
|---|---|
| author | Moure, Pehuén Pokel, Niclas Bounajma, Bilal Gao, Yingqiang Boehringer, Roman Cheng, Longbiao Liu, Shih-Chii |
| author_facet | Moure, Pehuén Pokel, Niclas Bounajma, Bilal Gao, Yingqiang Boehringer, Roman Cheng, Longbiao Liu, Shih-Chii |
| contents | Automatic speech recognition (ASR) systems remain brittle on dysarthric and other atypical speech. Recent audio-language models raise the possibility of improving performance by conditioning on additional clinical context at inference time, but it is unclear whether these models can make use of such information. We introduce a benchmark built on the Speech Accessibility Project (SAP) dataset that tests whether diagnosis labels, clinician-derived speech ratings, and progressively richer clinical descriptions improve transcription accuracy for dysarthric speech. Across matched comparisons on nine models, we find that current models do not meaningfully use this context: diagnosis-informed and clinically detailed prompts yield negligible improvements and often degrade word error rate. We complement the prompting analysis with context-dependent fine-tuning, showing that LoRA adaptation with a mixture of clinical prompt formats achieves a WER of 0.066, a 52% relative reduction over the frozen baseline, while preserving performance when context is unavailable. Subgroup analyses reveal significant gains for Down syndrome and mild-severity speakers. These results clarify where current models fall short and provide a testbed for measuring progress toward more inclusive ASR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_02782 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition Moure, Pehuén Pokel, Niclas Bounajma, Bilal Gao, Yingqiang Boehringer, Roman Cheng, Longbiao Liu, Shih-Chii Artificial Intelligence Computation and Language Audio and Speech Processing Automatic speech recognition (ASR) systems remain brittle on dysarthric and other atypical speech. Recent audio-language models raise the possibility of improving performance by conditioning on additional clinical context at inference time, but it is unclear whether these models can make use of such information. We introduce a benchmark built on the Speech Accessibility Project (SAP) dataset that tests whether diagnosis labels, clinician-derived speech ratings, and progressively richer clinical descriptions improve transcription accuracy for dysarthric speech. Across matched comparisons on nine models, we find that current models do not meaningfully use this context: diagnosis-informed and clinically detailed prompts yield negligible improvements and often degrade word error rate. We complement the prompting analysis with context-dependent fine-tuning, showing that LoRA adaptation with a mixture of clinical prompt formats achieves a WER of 0.066, a 52% relative reduction over the frozen baseline, while preserving performance when context is unavailable. Subgroup analyses reveal significant gains for Down syndrome and mild-severity speakers. These results clarify where current models fall short and provide a testbed for measuring progress toward more inclusive ASR. |
| title | When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition |
| topic | Artificial Intelligence Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2605.02782 |