Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913021643718656 |
|---|---|
| author | Li, Ziwei Dong, Lukuang Yusuyin, Saierdaer Zhao, Xianyu Ou, Zhijian |
| author_facet | Li, Ziwei Dong, Lukuang Yusuyin, Saierdaer Zhao, Xianyu Ou, Zhijian |
| contents | Integrating pretrained speech encoders with large language models (LLMs) is promising for ASR, but performance and data efficiency depend on the speech-language interface. A common choice is a learned projector that maps encoder features into the LLM embedding space, whereas an alternative is to expose discrete phoneme sequences to the LLM. Using the same encoder and LLM backbones, we compare phoneme-based and vanilla projector-based interfaces in high-resource English and low-resource Tatar. We also propose a BPE-phoneme interface that groups frequent local phoneme patterns while preserving explicit word-boundary cues for phoneme-to-grapheme generation. On LibriSpeech, the phoneme-based interface is competitive with the vanilla projector, and the BPE-phoneme interface yields further gains. On Tatar, the phoneme-based interface substantially outperforms the vanilla projector. We further find that phoneme supervision yields a phoneme-informed hybrid interface that is stronger than the vanilla projector. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_09332 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR Li, Ziwei Dong, Lukuang Yusuyin, Saierdaer Zhao, Xianyu Ou, Zhijian Audio and Speech Processing Integrating pretrained speech encoders with large language models (LLMs) is promising for ASR, but performance and data efficiency depend on the speech-language interface. A common choice is a learned projector that maps encoder features into the LLM embedding space, whereas an alternative is to expose discrete phoneme sequences to the LLM. Using the same encoder and LLM backbones, we compare phoneme-based and vanilla projector-based interfaces in high-resource English and low-resource Tatar. We also propose a BPE-phoneme interface that groups frequent local phoneme patterns while preserving explicit word-boundary cues for phoneme-to-grapheme generation. On LibriSpeech, the phoneme-based interface is competitive with the vanilla projector, and the BPE-phoneme interface yields further gains. On Tatar, the phoneme-based interface substantially outperforms the vanilla projector. We further find that phoneme supervision yields a phoneme-informed hybrid interface that is stronger than the vanilla projector. |
| title | Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2604.09332 |