FlowW2N: Whispered-to-Normal Speech Conversion via Flow-Matching
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910041111527424 |
|---|---|
| author | Ritter-Gutierrez, Fabian Jalal, Md Asif Parada, Pablo Peso Saravanan, Karthikeyan Shul, Yusun Kim, Minseung Lee, Gun-Woo Moon, Han-Gil |
| author_facet | Ritter-Gutierrez, Fabian Jalal, Md Asif Parada, Pablo Peso Saravanan, Karthikeyan Shul, Yusun Kim, Minseung Lee, Gun-Woo Moon, Han-Gil |
| contents | Whispered-to-normal (W2N) speech conversion aims to reconstruct missing phonation from whispered input while preserving content and speaker identity. This task is challenging due to temporal misalignment between whisper and voiced recordings and lack of paired data. We propose FlowW2N, a conditional flow matching approach that trains exclusively on synthetic, time-aligned whisper-normal pairs and conditions on domain-invariant features. We exploit high-level ASR embeddings that exhibits strong invariance between synthetic and real whispered speech, enabling generalization to real whispers despite never observing it during training. We verify this invariance across ASR layers and propose a selection criterion optimizing content informativeness and cross-domain invariance. Our method achieves SOTA intelligibility on the CHAINS and wTIMIT datasets, reducing Word Error Rate by 26-46% relative to prior work while using only 10 steps at inference and requiring no real paired data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_04296 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FlowW2N: Whispered-to-Normal Speech Conversion via Flow-Matching Ritter-Gutierrez, Fabian Jalal, Md Asif Parada, Pablo Peso Saravanan, Karthikeyan Shul, Yusun Kim, Minseung Lee, Gun-Woo Moon, Han-Gil Audio and Speech Processing Sound Whispered-to-normal (W2N) speech conversion aims to reconstruct missing phonation from whispered input while preserving content and speaker identity. This task is challenging due to temporal misalignment between whisper and voiced recordings and lack of paired data. We propose FlowW2N, a conditional flow matching approach that trains exclusively on synthetic, time-aligned whisper-normal pairs and conditions on domain-invariant features. We exploit high-level ASR embeddings that exhibits strong invariance between synthetic and real whispered speech, enabling generalization to real whispers despite never observing it during training. We verify this invariance across ASR layers and propose a selection criterion optimizing content informativeness and cross-domain invariance. Our method achieves SOTA intelligibility on the CHAINS and wTIMIT datasets, reducing Word Error Rate by 26-46% relative to prior work while using only 10 steps at inference and requiring no real paired data. |
| title | FlowW2N: Whispered-to-Normal Speech Conversion via Flow-Matching |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2603.04296 |