SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918532710662144 |
|---|---|
| author | Yegorova, Yekaterina Gerogiannis, Argyrios Zheng, Haolong Hockenmaier, Julia Yoo, Chang D. Hasegawa-Johnson, Mark A. |
| author_facet | Yegorova, Yekaterina Gerogiannis, Argyrios Zheng, Haolong Hockenmaier, Julia Yoo, Chang D. Hasegawa-Johnson, Mark A. |
| contents | Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children's speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained language model representation space, rather than by modifying the decoder itself. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_00460 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors Yegorova, Yekaterina Gerogiannis, Argyrios Zheng, Haolong Hockenmaier, Julia Yoo, Chang D. Hasegawa-Johnson, Mark A. Computation and Language Audio and Speech Processing Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children's speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained language model representation space, rather than by modifying the decoder itself. |
| title | SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2606.00460 |