A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910575969173504 |
|---|---|
| author | Watcharasupat, Karn N. Wu, Chih-Wei Ding, Yiwei Orife, Iroro Hipple, Aaron J. Williams, Phillip A. Kramer, Scott Lerch, Alexander Wolcott, William |
| author_facet | Watcharasupat, Karn N. Wu, Chih-Wei Ding, Yiwei Orife, Iroro Hipple, Aaron J. Williams, Phillip A. Kramer, Scott Lerch, Alexander Wolcott, William |
| contents | Cinematic audio source separation is a relatively new subtask of audio source separation, with the aim of extracting the dialogue, music, and effects stems from their mixture. In this work, we developed a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. Psychoacoustically motivated frequency scales were used to inform the band definitions which are now defined with redundancy for more reliable feature extraction. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. We additionally exploit the information-sharing property of a common-encoder setup to reduce computational complexity during both training and inference, improve separation performance for hard-to-generalize classes of sounds, and allow flexibility during inference time with detachable decoders. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_02539 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation Watcharasupat, Karn N. Wu, Chih-Wei Ding, Yiwei Orife, Iroro Hipple, Aaron J. Williams, Phillip A. Kramer, Scott Lerch, Alexander Wolcott, William Audio and Speech Processing Machine Learning Sound Signal Processing Cinematic audio source separation is a relatively new subtask of audio source separation, with the aim of extracting the dialogue, music, and effects stems from their mixture. In this work, we developed a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. Psychoacoustically motivated frequency scales were used to inform the band definitions which are now defined with redundancy for more reliable feature extraction. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. We additionally exploit the information-sharing property of a common-encoder setup to reduce computational complexity during both training and inference, improve separation performance for hard-to-generalize classes of sounds, and allow flexibility during inference time with detachable decoders. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem. |
| title | A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation |
| topic | Audio and Speech Processing Machine Learning Sound Signal Processing |
| url | https://arxiv.org/abs/2309.02539 |