Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912874249584640 |
|---|---|
| author | Kocour, Martin Karafiat, Martin Polok, Alexander Klement, Dominik Burget, Lukáš Černocký, Jan |
| author_facet | Kocour, Martin Karafiat, Martin Polok, Alexander Klement, Dominik Burget, Lukáš Černocký, Jan |
| contents | We propose a speaker-attributed (SA) Whisper-based model for multi-talker speech recognition that combines target-speaker modeling with serialized output training (SOT). Our approach leverages a Diarization-Conditioned Whisper (DiCoW) encoder to extract target-speaker embeddings, which are concatenated into a single representation and passed to a shared decoder. This enables the model to transcribe overlapping speech as a serialized output stream with speaker tags and timestamps. In contrast to target-speaker ASR systems such as DiCoW, which decode each speaker separately, our approach performs joint decoding, allowing the decoder to condition on the context of all speakers simultaneously. Experiments show that the model outperforms existing SOT-based approaches and surpasses DiCoW on multi-talker mixtures (e.g., LibriMix). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_03723 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition Kocour, Martin Karafiat, Martin Polok, Alexander Klement, Dominik Burget, Lukáš Černocký, Jan Audio and Speech Processing Computation and Language Sound We propose a speaker-attributed (SA) Whisper-based model for multi-talker speech recognition that combines target-speaker modeling with serialized output training (SOT). Our approach leverages a Diarization-Conditioned Whisper (DiCoW) encoder to extract target-speaker embeddings, which are concatenated into a single representation and passed to a shared decoder. This enables the model to transcribe overlapping speech as a serialized output stream with speaker tags and timestamps. In contrast to target-speaker ASR systems such as DiCoW, which decode each speaker separately, our approach performs joint decoding, allowing the decoder to condition on the context of all speakers simultaneously. Experiments show that the model outperforms existing SOT-based approaches and surpasses DiCoW on multi-talker mixtures (e.g., LibriMix). |
| title | Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2510.03723 |