Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911678066589696 |
|---|---|
| author | Shakeel, Muhammad Sudo, Yui Peng, Yifan Lin, Chyi-Jiunn Watanabe, Shinji |
| author_facet | Shakeel, Muhammad Sudo, Yui Peng, Yifan Lin, Chyi-Jiunn Watanabe, Shinji |
| contents | This paper presents a unified multi-speaker encoder (UME), a novel architecture that jointly learns representations for speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) tasks using a shared speech foundational encoder. We leverage the hidden representations from multiple layers of UME as a residual weighted-sum encoding (RWSE) to effectively use information from different semantic levels, contributing to bottom-up alignment between tasks. This joint training approach captures the inherent interdependencies among the tasks, enhancing overall performance on overlapping speech data. Our evaluations demonstrate that UME substantially improves over the single-task baselines dedicated to SD, SS, and multi-speaker ASR on LibriMix evaluation sets. Notably, for SD, UME outperforms the previous studies, achieving diarization error rates of 1.37% and 2.29% on Libri2Mix and Libri3Mix evaluation sets, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_20474 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder Shakeel, Muhammad Sudo, Yui Peng, Yifan Lin, Chyi-Jiunn Watanabe, Shinji Audio and Speech Processing Computation and Language Sound This paper presents a unified multi-speaker encoder (UME), a novel architecture that jointly learns representations for speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) tasks using a shared speech foundational encoder. We leverage the hidden representations from multiple layers of UME as a residual weighted-sum encoding (RWSE) to effectively use information from different semantic levels, contributing to bottom-up alignment between tasks. This joint training approach captures the inherent interdependencies among the tasks, enhancing overall performance on overlapping speech data. Our evaluations demonstrate that UME substantially improves over the single-task baselines dedicated to SD, SS, and multi-speaker ASR on LibriMix evaluation sets. Notably, for SD, UME outperforms the previous studies, achieving diarization error rates of 1.37% and 2.29% on Libri2Mix and Libri3Mix evaluation sets, respectively. |
| title | Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2508.20474 |