Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.17902 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916336083402752 |
|---|---|
| author | Tao, Ruijie Shi, Zhan Jiang, Yidi Truong, Duc-Tuan Chng, Eng-Siong Alioto, Massimo Li, Haizhou |
| author_facet | Tao, Ruijie Shi, Zhan Jiang, Yidi Truong, Duc-Tuan Chng, Eng-Siong Alioto, Massimo Li, Haizhou |
| contents | The human brain has the capability to associate the unknown person's voice and face by leveraging their general relationship, referred to as ``cross-modal speaker verification''. This task poses significant challenges due to the complex relationship between the modalities. In this paper, we propose a ``Multi-stage Face-voice Association Learning with Keynote Speaker Diarization''~(MFV-KSD) framework. MFV-KSD contains a keynote speaker diarization front-end to effectively address the noisy speech inputs issue. To balance and enhance the intra-modal feature learning and inter-modal correlation understanding, MFV-KSD utilizes a novel three-stage training strategy. Our experimental results demonstrated robust performance, achieving the first rank in the 2024 Face-voice Association in Multilingual Environments (FAME) challenge with an overall Equal Error Rate (EER) of 19.9%. Details can be found in https://github.com/TaoRuijie/MFV-KSD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_17902 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-Stage Face-Voice Association Learning with Keynote Speaker Diarization Tao, Ruijie Shi, Zhan Jiang, Yidi Truong, Duc-Tuan Chng, Eng-Siong Alioto, Massimo Li, Haizhou Audio and Speech Processing The human brain has the capability to associate the unknown person's voice and face by leveraging their general relationship, referred to as ``cross-modal speaker verification''. This task poses significant challenges due to the complex relationship between the modalities. In this paper, we propose a ``Multi-stage Face-voice Association Learning with Keynote Speaker Diarization''~(MFV-KSD) framework. MFV-KSD contains a keynote speaker diarization front-end to effectively address the noisy speech inputs issue. To balance and enhance the intra-modal feature learning and inter-modal correlation understanding, MFV-KSD utilizes a novel three-stage training strategy. Our experimental results demonstrated robust performance, achieving the first rank in the 2024 Face-voice Association in Multilingual Environments (FAME) challenge with an overall Equal Error Rate (EER) of 19.9%. Details can be found in https://github.com/TaoRuijie/MFV-KSD. |
| title | Multi-Stage Face-Voice Association Learning with Keynote Speaker Diarization |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2407.17902 |