Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2606.00460 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918532710662144 |
|---|---|
| author | Yegorova, Yekaterina Gerogiannis, Argyrios Zheng, Haolong Hockenmaier, Julia Yoo, Chang D. Hasegawa-Johnson, Mark A. |
| author_facet | Yegorova, Yekaterina Gerogiannis, Argyrios Zheng, Haolong Hockenmaier, Julia Yoo, Chang D. Hasegawa-Johnson, Mark A. |
| contents | Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children's speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained language model representation space, rather than by modifying the decoder itself. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_00460 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors Yegorova, Yekaterina Gerogiannis, Argyrios Zheng, Haolong Hockenmaier, Julia Yoo, Chang D. Hasegawa-Johnson, Mark A. Computation and Language Audio and Speech Processing Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children's speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained language model representation space, rather than by modifying the decoder itself. |
| title | SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2606.00460 |