An Empirical Study on the Impact of Positional Encoding in Transformer-based Monaural Speech Enhancement
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914677796110336 |
|---|---|
| author | Zhang, Qiquan Ge, Meng Zhu, Hongxu Ambikairajah, Eliathamby Song, Qi Ni, Zhaoheng Li, Haizhou |
| author_facet | Zhang, Qiquan Ge, Meng Zhu, Hongxu Ambikairajah, Eliathamby Song, Qi Ni, Zhaoheng Li, Haizhou |
| contents | Transformer architecture has enabled recent progress in speech enhancement. Since Transformers are position-agostic, positional encoding is the de facto standard component used to enable Transformers to distinguish the order of elements in a sequence. However, it remains unclear how positional encoding exactly impacts speech enhancement based on Transformer architectures. In this paper, we perform a comprehensive empirical study evaluating five positional encoding methods, i.e., Sinusoidal and learned absolute position embedding (APE), T5-RPE, KERPLE, as well as the Transformer without positional encoding (No-Pos), across both causal and noncausal configurations. We conduct extensive speech enhancement experiments, involving spectral mapping and masking methods. Our findings establish that positional encoding is not quite helpful for the models in a causal configuration, which indicates that causal attention may implicitly incorporate position information. In a noncausal configuration, the models significantly benefit from the use of positional encoding. In addition, we find that among the four position embeddings, relative position embeddings outperform APEs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_09686 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Empirical Study on the Impact of Positional Encoding in Transformer-based Monaural Speech Enhancement Zhang, Qiquan Ge, Meng Zhu, Hongxu Ambikairajah, Eliathamby Song, Qi Ni, Zhaoheng Li, Haizhou Audio and Speech Processing Sound Transformer architecture has enabled recent progress in speech enhancement. Since Transformers are position-agostic, positional encoding is the de facto standard component used to enable Transformers to distinguish the order of elements in a sequence. However, it remains unclear how positional encoding exactly impacts speech enhancement based on Transformer architectures. In this paper, we perform a comprehensive empirical study evaluating five positional encoding methods, i.e., Sinusoidal and learned absolute position embedding (APE), T5-RPE, KERPLE, as well as the Transformer without positional encoding (No-Pos), across both causal and noncausal configurations. We conduct extensive speech enhancement experiments, involving spectral mapping and masking methods. Our findings establish that positional encoding is not quite helpful for the models in a causal configuration, which indicates that causal attention may implicitly incorporate position information. In a noncausal configuration, the models significantly benefit from the use of positional encoding. In addition, we find that among the four position embeddings, relative position embeddings outperform APEs. |
| title | An Empirical Study on the Impact of Positional Encoding in Transformer-based Monaural Speech Enhancement |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2401.09686 |