Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2409.02266 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916607496814592 |
|---|---|
| author | Jain, Arnav Sanjotra, Jasmer Singh Choudhary, Harshvardhan Agrawal, Krish Shah, Rupal Jha, Rohan Sajid, M. Hussain, Amir Tanveer, M. |
| author_facet | Jain, Arnav Sanjotra, Jasmer Singh Choudhary, Harshvardhan Agrawal, Krish Shah, Rupal Jha, Rohan Sajid, M. Hussain, Amir Tanveer, M. |
| contents | In this paper, we propose long short term memory speech enhancement network (LSTMSE-Net), an audio-visual speech enhancement (AVSE) method. This innovative method leverages the complementary nature of visual and audio information to boost the quality of speech signals. Visual features are extracted with VisualFeatNet (VFN), and audio features are processed through an encoder and decoder. The system scales and concatenates visual and audio features, then processes them through a separator network for optimized speech enhancement. The architecture highlights advancements in leveraging multi-modal data and interpolation techniques for robust AVSE challenge systems. The performance of LSTMSE-Net surpasses that of the baseline model from the COG-MHEAR AVSE Challenge 2024 by a margin of 0.06 in scale-invariant signal-to-distortion ratio (SISDR), $0.03$ in short-time objective intelligibility (STOI), and $1.32$ in perceptual evaluation of speech quality (PESQ). The source code of the proposed LSTMSE-Net is available at \url{https://github.com/mtanveer1/AVSEC-3-Challenge}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_02266 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LSTMSE-Net: Long Short Term Speech Enhancement Network for Audio-visual Speech Enhancement Jain, Arnav Sanjotra, Jasmer Singh Choudhary, Harshvardhan Agrawal, Krish Shah, Rupal Jha, Rohan Sajid, M. Hussain, Amir Tanveer, M. Sound Machine Learning Multimedia Audio and Speech Processing In this paper, we propose long short term memory speech enhancement network (LSTMSE-Net), an audio-visual speech enhancement (AVSE) method. This innovative method leverages the complementary nature of visual and audio information to boost the quality of speech signals. Visual features are extracted with VisualFeatNet (VFN), and audio features are processed through an encoder and decoder. The system scales and concatenates visual and audio features, then processes them through a separator network for optimized speech enhancement. The architecture highlights advancements in leveraging multi-modal data and interpolation techniques for robust AVSE challenge systems. The performance of LSTMSE-Net surpasses that of the baseline model from the COG-MHEAR AVSE Challenge 2024 by a margin of 0.06 in scale-invariant signal-to-distortion ratio (SISDR), $0.03$ in short-time objective intelligibility (STOI), and $1.32$ in perceptual evaluation of speech quality (PESQ). The source code of the proposed LSTMSE-Net is available at \url{https://github.com/mtanveer1/AVSEC-3-Challenge}. |
| title | LSTMSE-Net: Long Short Term Speech Enhancement Network for Audio-visual Speech Enhancement |
| topic | Sound Machine Learning Multimedia Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.02266 |