Towards Sub-millisecond Latency Real-Time Speech Enhancement Models on Hearables
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929748106543104 |
|---|---|
| author | Dementyev, Artem Reddy, Chandan K. A. Wisdom, Scott Chatlani, Navin Hershey, John R. Lyon, Richard F. |
| author_facet | Dementyev, Artem Reddy, Chandan K. A. Wisdom, Scott Chatlani, Navin Hershey, John R. Lyon, Richard F. |
| contents | Low latency models are critical for real-time speech enhancement applications, such as hearing aids and hearables. However, the sub-millisecond latency space for resource-constrained hearables remains underexplored. We demonstrate speech enhancement using a computationally efficient minimum-phase FIR filter, enabling sample-by-sample processing to achieve mean algorithmic latency of 0.32 ms to 1.25 ms. With a single microphone, we observe a mean SI-SDRi of 4.1 dB. The approach shows generalization with a DNSMOS increase of 0.2 on unseen audio recordings. We use a lightweight LSTM-based model of 626k parameters to generate FIR taps. Using a real hardware implementation on a low-power DSP, our system can run with 376 MIPS and a mean end-to-end latency of 3.35 ms. In addition, we provide a comparison with existing low-latency spectral masking techniques. We hope this work will enable a better understanding of latency and can be used to improve the comfort and usability of hearables. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18239 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Sub-millisecond Latency Real-Time Speech Enhancement Models on Hearables Dementyev, Artem Reddy, Chandan K. A. Wisdom, Scott Chatlani, Navin Hershey, John R. Lyon, Richard F. Sound Machine Learning Audio and Speech Processing Low latency models are critical for real-time speech enhancement applications, such as hearing aids and hearables. However, the sub-millisecond latency space for resource-constrained hearables remains underexplored. We demonstrate speech enhancement using a computationally efficient minimum-phase FIR filter, enabling sample-by-sample processing to achieve mean algorithmic latency of 0.32 ms to 1.25 ms. With a single microphone, we observe a mean SI-SDRi of 4.1 dB. The approach shows generalization with a DNSMOS increase of 0.2 on unseen audio recordings. We use a lightweight LSTM-based model of 626k parameters to generate FIR taps. Using a real hardware implementation on a low-power DSP, our system can run with 376 MIPS and a mean end-to-end latency of 3.35 ms. In addition, we provide a comparison with existing low-latency spectral masking techniques. We hope this work will enable a better understanding of latency and can be used to improve the comfort and usability of hearables. |
| title | Towards Sub-millisecond Latency Real-Time Speech Enhancement Models on Hearables |
| topic | Sound Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.18239 |