Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.23284 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911622989086720 |
|---|---|
| author | Liu, Meizhu Mitra, Nistha Li, Paul Abdaoui, Amine Ledyard, Adam Sheng, Tao |
| author_facet | Liu, Meizhu Mitra, Nistha Li, Paul Abdaoui, Amine Ledyard, Adam Sheng, Tao |
| contents | In this work, we present Au-M-ol, a novel multimodal architecture that extends Large Language Models (LLMs) with audio processing. It is designed to improve performance on clinically relevant tasks such as Automatic Speech Recognition (ASR). Au-M-ol has three main components: (1) an audio encoder that extracts rich acoustic features from medical speech, (2) an adaptation layer that maps audio features into the LLM input space, and (3) a pretrained LLM that performs transcription and clinical language understanding. This design allows the model to interpret spoken medical content directly, improving both accuracy and robustness. In experiments, Au-M-ol reduces Word Error Rate (WER) by 56\% compared to state-of-the-art baselines on medical transcription tasks. The model also performs well in challenging conditions, including noisy environments, domain-specific terminology, and speaker variability. These results suggest that Au-M-ol is a strong candidate for real-world clinical applications, where reliable and context-aware audio understanding is essential. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_23284 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Au-M-ol: A Unified Model for Medical Audio and Language Understanding Liu, Meizhu Mitra, Nistha Li, Paul Abdaoui, Amine Ledyard, Adam Sheng, Tao Computation and Language Artificial Intelligence In this work, we present Au-M-ol, a novel multimodal architecture that extends Large Language Models (LLMs) with audio processing. It is designed to improve performance on clinically relevant tasks such as Automatic Speech Recognition (ASR). Au-M-ol has three main components: (1) an audio encoder that extracts rich acoustic features from medical speech, (2) an adaptation layer that maps audio features into the LLM input space, and (3) a pretrained LLM that performs transcription and clinical language understanding. This design allows the model to interpret spoken medical content directly, improving both accuracy and robustness. In experiments, Au-M-ol reduces Word Error Rate (WER) by 56\% compared to state-of-the-art baselines on medical transcription tasks. The model also performs well in challenging conditions, including noisy environments, domain-specific terminology, and speaker variability. These results suggest that Au-M-ol is a strong candidate for real-world clinical applications, where reliable and context-aware audio understanding is essential. |
| title | Au-M-ol: A Unified Model for Medical Audio and Language Understanding |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.23284 |