Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.19606 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918426862157824 |
|---|---|
| author | Shim, Ryan Soh-Eun De Cristofaro, Domenico Hu, Chengzhi Martin Vietti, Alessandro Plank, Barbara |
| author_facet | Shim, Ryan Soh-Eun De Cristofaro, Domenico Hu, Chengzhi Martin Vietti, Alessandro Plank, Barbara |
| contents | Cross-lingual alignment in pretrained language models enables knowledge transfer across languages. Similar alignment has been reported in Whisper-style speech encoders, based on spoken translation retrieval using representational similarity. However, prior work does not control for phonetic overlap between equivalent utterances, which may artificially support retrieval. We conduct pronunciation-controlled experiments to test whether cross-lingual alignment arises from semantic rather than phonetic similarity. Results show that spoken translation retrieval remains strongly above chance without phonetic cues in the final layers of encoders trained with a speech translation objective, most clearly for models additionally trained on translation. We further test early-exiting the encoder to induce representations we hypothesize to be less tied to language-specific semantics. These experiments indeed reveal performance gains in automatic speech recognition on low-resource languages unseen during training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19606 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Languages in Whisper-Style Speech Encoders Align Both Phonetically and Semantically Shim, Ryan Soh-Eun De Cristofaro, Domenico Hu, Chengzhi Martin Vietti, Alessandro Plank, Barbara Computation and Language Cross-lingual alignment in pretrained language models enables knowledge transfer across languages. Similar alignment has been reported in Whisper-style speech encoders, based on spoken translation retrieval using representational similarity. However, prior work does not control for phonetic overlap between equivalent utterances, which may artificially support retrieval. We conduct pronunciation-controlled experiments to test whether cross-lingual alignment arises from semantic rather than phonetic similarity. Results show that spoken translation retrieval remains strongly above chance without phonetic cues in the final layers of encoders trained with a speech translation objective, most clearly for models additionally trained on translation. We further test early-exiting the encoder to induce representations we hypothesize to be less tied to language-specific semantics. These experiments indeed reveal performance gains in automatic speech recognition on low-resource languages unseen during training. |
| title | Languages in Whisper-Style Speech Encoders Align Both Phonetically and Semantically |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.19606 |