DiffEditor: Enhancing Speech Editing with Semantic Enrichment and Acoustic Consistency
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910613649752064 |
|---|---|
| author | Chen, Yang Jia, Yuhang Zhao, Shiwan Jiang, Ziyue Li, Haoran Kang, Jiarong Qin, Yong |
| author_facet | Chen, Yang Jia, Yuhang Zhao, Shiwan Jiang, Ziyue Li, Haoran Kang, Jiarong Qin, Yong |
| contents | As text-based speech editing becomes increasingly prevalent, the demand for unrestricted free-text editing continues to grow. However, existing speech editing techniques encounter significant challenges, particularly in maintaining intelligibility and acoustic consistency when dealing with out-of-domain (OOD) text. In this paper, we introduce, DiffEditor, a novel speech editing model designed to enhance performance in OOD text scenarios through semantic enrichment and acoustic consistency. To improve the intelligibility of the edited speech, we enrich the semantic information of phoneme embeddings by integrating word embeddings extracted from a pretrained language model. Furthermore, we emphasize that interframe smoothing properties are critical for modeling acoustic consistency, and thus we propose a first-order loss function to promote smoother transitions at editing boundaries and enhance the overall fluency of the edited speech. Experimental results demonstrate that our model achieves state-of-the-art performance in both in-domain and OOD text scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12992 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DiffEditor: Enhancing Speech Editing with Semantic Enrichment and Acoustic Consistency Chen, Yang Jia, Yuhang Zhao, Shiwan Jiang, Ziyue Li, Haoran Kang, Jiarong Qin, Yong Sound Artificial Intelligence Machine Learning Audio and Speech Processing As text-based speech editing becomes increasingly prevalent, the demand for unrestricted free-text editing continues to grow. However, existing speech editing techniques encounter significant challenges, particularly in maintaining intelligibility and acoustic consistency when dealing with out-of-domain (OOD) text. In this paper, we introduce, DiffEditor, a novel speech editing model designed to enhance performance in OOD text scenarios through semantic enrichment and acoustic consistency. To improve the intelligibility of the edited speech, we enrich the semantic information of phoneme embeddings by integrating word embeddings extracted from a pretrained language model. Furthermore, we emphasize that interframe smoothing properties are critical for modeling acoustic consistency, and thus we propose a first-order loss function to promote smoother transitions at editing boundaries and enhance the overall fluency of the edited speech. Experimental results demonstrate that our model achieves state-of-the-art performance in both in-domain and OOD text scenarios. |
| title | DiffEditor: Enhancing Speech Editing with Semantic Enrichment and Acoustic Consistency |
| topic | Sound Artificial Intelligence Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.12992 |