MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915941255741440 |
|---|---|
| author | Takahashi, Akira Takahashi, Shusuke Mitsufuji, Yuki |
| author_facet | Takahashi, Akira Takahashi, Shusuke Mitsufuji, Yuki |
| contents | We introduce MMAudioSep, a generative model for video/text-queried sound separation that is founded on a pretrained video-to-audio model. By leveraging knowledge about the relationship between video/text and audio learned through a pretrained audio generative model, we can train the model more efficiently, i.e., the model does not need to be trained from scratch. We evaluate the performance of MMAudioSep by comparing it to existing separation models, including models based on both deterministic and generative approaches, and find it is superior to the baseline models. Furthermore, we demonstrate that even after acquiring functionality for sound separation via fine-tuning, the model retains the ability for original video-to-audio generation. This highlights the potential of foundational sound generation models to be adopted for sound-related downstream tasks. Our code is available at https://github.com/sony/mmaudiosep. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09065 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation Takahashi, Akira Takahashi, Shusuke Mitsufuji, Yuki Sound Computer Vision and Pattern Recognition Machine Learning Audio and Speech Processing We introduce MMAudioSep, a generative model for video/text-queried sound separation that is founded on a pretrained video-to-audio model. By leveraging knowledge about the relationship between video/text and audio learned through a pretrained audio generative model, we can train the model more efficiently, i.e., the model does not need to be trained from scratch. We evaluate the performance of MMAudioSep by comparing it to existing separation models, including models based on both deterministic and generative approaches, and find it is superior to the baseline models. Furthermore, we demonstrate that even after acquiring functionality for sound separation via fine-tuning, the model retains the ability for original video-to-audio generation. This highlights the potential of foundational sound generation models to be adopted for sound-related downstream tasks. Our code is available at https://github.com/sony/mmaudiosep. |
| title | MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation |
| topic | Sound Computer Vision and Pattern Recognition Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2510.09065 |