PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915028302561280 |
|---|---|
| author | Kalda, Joonas Pagés, Clément Marxer, Ricard Alumäe, Tanel Bredin, Hervé |
| author_facet | Kalda, Joonas Pagés, Clément Marxer, Ricard Alumäe, Tanel Bredin, Hervé |
| contents | A major drawback of supervised speech separation (SSep) systems is their reliance on synthetic data, leading to poor real-world generalization. Mixture invariant training (MixIT) was proposed as an unsupervised alternative that uses real recordings, yet struggles with overseparation and adapting to long-form audio. We introduce PixIT, a joint approach that combines permutation invariant training (PIT) for speaker diarization (SD) and MixIT for SSep. With a small extra requirement of needing SD labels, it solves the problem of overseparation and allows stitching local separated sources leveraging existing work on clustering-based neural SD. We measure the quality of the separated sources via applying automatic speech recognition (ASR) systems to them. PixIT boosts the performance of various ASR systems across two meeting corpora both in terms of the speaker-attributed and utterance-based word error rates while not requiring any fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_02288 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings Kalda, Joonas Pagés, Clément Marxer, Ricard Alumäe, Tanel Bredin, Hervé Audio and Speech Processing A major drawback of supervised speech separation (SSep) systems is their reliance on synthetic data, leading to poor real-world generalization. Mixture invariant training (MixIT) was proposed as an unsupervised alternative that uses real recordings, yet struggles with overseparation and adapting to long-form audio. We introduce PixIT, a joint approach that combines permutation invariant training (PIT) for speaker diarization (SD) and MixIT for SSep. With a small extra requirement of needing SD labels, it solves the problem of overseparation and allows stitching local separated sources leveraging existing work on clustering-based neural SD. We measure the quality of the separated sources via applying automatic speech recognition (ASR) systems to them. PixIT boosts the performance of various ASR systems across two meeting corpora both in terms of the speaker-attributed and utterance-based word error rates while not requiring any fine-tuning. |
| title | PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2403.02288 |