ReverbMiipher: Generative Speech Restoration meets Reverberation Characteristics Controllability
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915389670162432 |
|---|---|
| author | Nakata, Wataru Koizumi, Yuma Karita, Shigeki Scheibler, Robin Ishikawa, Haruko Guevara-Rukoz, Adriana Zen, Heiga Bacchiani, Michiel |
| author_facet | Nakata, Wataru Koizumi, Yuma Karita, Shigeki Scheibler, Robin Ishikawa, Haruko Guevara-Rukoz, Adriana Zen, Heiga Bacchiani, Michiel |
| contents | Reverberation encodes spatial information regarding the acoustic source environment, yet traditional Speech Restoration (SR) usually completely removes reverberation. We propose ReverbMiipher, an SR model extending parametric resynthesis framework, designed to denoise speech while preserving and enabling control over reverberation. ReverbMiipher incorporates a dedicated ReverbEncoder to extract a reverb feature vector from noisy input. This feature conditions a vocoder to reconstruct the speech signal, removing noise while retaining the original reverberation characteristics. A stochastic zero-vector replacement strategy during training ensures the feature specifically encodes reverberation, disentangling it from other speech attributes. This learned representation facilitates reverberation control via techniques such as interpolation between features, replacement with features from other utterances, or sampling from a latent space. Objective and subjective evaluations confirm ReverbMiipher effectively preserves reverberation, removes other artifacts, and outperforms the conventional two-stage SR and convolving simulated room impulse response approach. We further demonstrate its ability to generate novel reverberation effects through feature manipulation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_05077 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ReverbMiipher: Generative Speech Restoration meets Reverberation Characteristics Controllability Nakata, Wataru Koizumi, Yuma Karita, Shigeki Scheibler, Robin Ishikawa, Haruko Guevara-Rukoz, Adriana Zen, Heiga Bacchiani, Michiel Sound Audio and Speech Processing Reverberation encodes spatial information regarding the acoustic source environment, yet traditional Speech Restoration (SR) usually completely removes reverberation. We propose ReverbMiipher, an SR model extending parametric resynthesis framework, designed to denoise speech while preserving and enabling control over reverberation. ReverbMiipher incorporates a dedicated ReverbEncoder to extract a reverb feature vector from noisy input. This feature conditions a vocoder to reconstruct the speech signal, removing noise while retaining the original reverberation characteristics. A stochastic zero-vector replacement strategy during training ensures the feature specifically encodes reverberation, disentangling it from other speech attributes. This learned representation facilitates reverberation control via techniques such as interpolation between features, replacement with features from other utterances, or sampling from a latent space. Objective and subjective evaluations confirm ReverbMiipher effectively preserves reverberation, removes other artifacts, and outperforms the conventional two-stage SR and convolving simulated room impulse response approach. We further demonstrate its ability to generate novel reverberation effects through feature manipulation. |
| title | ReverbMiipher: Generative Speech Restoration meets Reverberation Characteristics Controllability |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2505.05077 |