Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913109027848192 |
|---|---|
| author | Burdisso, Sergio Villatoro-Tello, Esaú Kumar, Shashi Madikeri, Srikanth Carofilis, Andrés Rangappa, Pradeep E, Manjunath K Hacioglu, Kadri Motlicek, Petr Stolcke, Andreas |
| author_facet | Burdisso, Sergio Villatoro-Tello, Esaú Kumar, Shashi Madikeri, Srikanth Carofilis, Andrés Rangappa, Pradeep E, Manjunath K Hacioglu, Kadri Motlicek, Petr Stolcke, Andreas |
| contents | LLM-based automatic speech recognition (ASR), a well-established approach, connects speech foundation models to large language models (LLMs) through a speech-to-LLM projector, yielding promising results. A common design choice in these architectures is the use of a fixed, manually defined prompt during both training and inference. This setup not only enables applicability across a range of practical scenarios, but also helps maximize model performance. However, the impact of prompt design remains underexplored. This paper presents a comprehensive analysis of commonly used prompts across diverse datasets, showing that prompt choice significantly affects ASR performance and introduces instability, with no single prompt performing best across all cases. Inspired by the speech-to-LLM projector, we propose a prompt projector module, a simple, model-agnostic extension that learns to project prompt embeddings to more effective regions of the LLM input space, without modifying the underlying LLM-based ASR model. Experiments on four datasets show that the addition of a prompt projector consistently improves performance, reduces variability, and outperforms the best manually selected prompts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_20898 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection Burdisso, Sergio Villatoro-Tello, Esaú Kumar, Shashi Madikeri, Srikanth Carofilis, Andrés Rangappa, Pradeep E, Manjunath K Hacioglu, Kadri Motlicek, Petr Stolcke, Andreas Audio and Speech Processing Computation and Language Machine Learning LLM-based automatic speech recognition (ASR), a well-established approach, connects speech foundation models to large language models (LLMs) through a speech-to-LLM projector, yielding promising results. A common design choice in these architectures is the use of a fixed, manually defined prompt during both training and inference. This setup not only enables applicability across a range of practical scenarios, but also helps maximize model performance. However, the impact of prompt design remains underexplored. This paper presents a comprehensive analysis of commonly used prompts across diverse datasets, showing that prompt choice significantly affects ASR performance and introduces instability, with no single prompt performing best across all cases. Inspired by the speech-to-LLM projector, we propose a prompt projector module, a simple, model-agnostic extension that learns to project prompt embeddings to more effective regions of the LLM input space, without modifying the underlying LLM-based ASR model. Experiments on four datasets show that the addition of a prompt projector consistently improves performance, reduces variability, and outperforms the best manually selected prompts. |
| title | Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection |
| topic | Audio and Speech Processing Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2601.20898 |