Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.10587 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918383217278976 |
|---|---|
| author | Shi, Hao Fujita, Yusuke Koshkin, Roman Zhao, Mengjie Gao, Yuan Liu, Lianbo Sudo, Yui |
| author_facet | Shi, Hao Fujita, Yusuke Koshkin, Roman Zhao, Mengjie Gao, Yuan Liu, Lianbo Sudo, Yui |
| contents | Large language models (LLMs) provide strong semantic priors that can improve multi-talker automatic speech recognition (MT-ASR), but using an LLM as an autoregressive decoder is computationally expensive and remains fragile under heavy overlap. In this paper, we propose an encoder-only MT-ASR framework that adapts an LLM to multi-talker conditioning and distills its semantic guidance into the encoder during training, while retaining fast CTC-style decoding at inference. Our model employs a post-encoder separator with serialized CTC to produce talker-ordered transcripts, and leverages an adapted LLM-based SOT objective as a multi-talker-aware teacher signal to explicitly regularize mixed-speech representations. To further support variable numbers of talkers, we introduce a Talker-Count Head that predicts the talker count and dynamically selects the appropriate decoding branch. Experiments on LibriMix show that the proposed encoder-only model achieves comparable performance to LLM-based systems in the two-talker condition, while delivering significant improvements in the three-talker condition with significant small RTF. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_10587 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing Shi, Hao Fujita, Yusuke Koshkin, Roman Zhao, Mengjie Gao, Yuan Liu, Lianbo Sudo, Yui Sound Large language models (LLMs) provide strong semantic priors that can improve multi-talker automatic speech recognition (MT-ASR), but using an LLM as an autoregressive decoder is computationally expensive and remains fragile under heavy overlap. In this paper, we propose an encoder-only MT-ASR framework that adapts an LLM to multi-talker conditioning and distills its semantic guidance into the encoder during training, while retaining fast CTC-style decoding at inference. Our model employs a post-encoder separator with serialized CTC to produce talker-ordered transcripts, and leverages an adapted LLM-based SOT objective as a multi-talker-aware teacher signal to explicitly regularize mixed-speech representations. To further support variable numbers of talkers, we introduce a Talker-Count Head that predicts the talker count and dynamically selects the appropriate decoding branch. Experiments on LibriMix show that the proposed encoder-only model achieves comparable performance to LLM-based systems in the two-talker condition, while delivering significant improvements in the three-talker condition with significant small RTF. |
| title | Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing |
| topic | Sound |
| url | https://arxiv.org/abs/2603.10587 |