SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight Conv-TasNet and State Space Modeling
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910509977042944 |
|---|---|
| author | Sato, Hiroshi Moriya, Takafumi Mimura, Masato Horiguchi, Shota Ochiai, Tsubasa Ashihara, Takanori Ando, Atsushi Shinayama, Kentaro Delcroix, Marc |
| author_facet | Sato, Hiroshi Moriya, Takafumi Mimura, Masato Horiguchi, Shota Ochiai, Tsubasa Ashihara, Takanori Ando, Atsushi Shinayama, Kentaro Delcroix, Marc |
| contents | Real-time target speaker extraction (TSE) is intended to extract the desired speaker's voice from the observed mixture of multiple speakers in a streaming manner. Implementing real-time TSE is challenging as the computational complexity must be reduced to provide real-time operation. This work introduces to Conv-TasNet-based TSE a new architecture based on state space modeling (SSM) that has been shown to model long-term dependency effectively. Owing to SSM, fewer dilated convolutional layers are required to capture temporal dependency in Conv-TasNet, resulting in the reduction of model complexity. We also enlarge the window length and shift of the convolutional (TasNet) frontend encoder to reduce the computational cost further; the performance decline is compensated by over-parameterization of the frontend encoder. The proposed method reduces the real-time factor by 78% from the conventional causal Conv-TasNet-based TSE while matching its performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01857 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight Conv-TasNet and State Space Modeling Sato, Hiroshi Moriya, Takafumi Mimura, Masato Horiguchi, Shota Ochiai, Tsubasa Ashihara, Takanori Ando, Atsushi Shinayama, Kentaro Delcroix, Marc Audio and Speech Processing Sound Signal Processing Real-time target speaker extraction (TSE) is intended to extract the desired speaker's voice from the observed mixture of multiple speakers in a streaming manner. Implementing real-time TSE is challenging as the computational complexity must be reduced to provide real-time operation. This work introduces to Conv-TasNet-based TSE a new architecture based on state space modeling (SSM) that has been shown to model long-term dependency effectively. Owing to SSM, fewer dilated convolutional layers are required to capture temporal dependency in Conv-TasNet, resulting in the reduction of model complexity. We also enlarge the window length and shift of the convolutional (TasNet) frontend encoder to reduce the computational cost further; the performance decline is compensated by over-parameterization of the frontend encoder. The proposed method reduces the real-time factor by 78% from the conventional causal Conv-TasNet-based TSE while matching its performance. |
| title | SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight Conv-TasNet and State Space Modeling |
| topic | Audio and Speech Processing Sound Signal Processing |
| url | https://arxiv.org/abs/2407.01857 |