Mel-Refine: A Plug-and-Play Approach to Refine Mel-Spectrogram in Audio Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915059226116096 |
|---|---|
| author | Guo, Hongming Fu, Ruibo Geng, Yizhong Liu, Shuai Shi, Shuchen Wang, Tao Qiang, Chunyu Li, Chenxing Li, Ya Wen, Zhengqi Liu, Yukun Liu, Xuefei |
| author_facet | Guo, Hongming Fu, Ruibo Geng, Yizhong Liu, Shuai Shi, Shuchen Wang, Tao Qiang, Chunyu Li, Chenxing Li, Ya Wen, Zhengqi Liu, Yukun Liu, Xuefei |
| contents | Text-to-audio (TTA) model is capable of generating diverse audio from textual prompts. However, most mainstream TTA models, which predominantly rely on Mel-spectrograms, still face challenges in producing audio with rich content. The intricate details and texture required in Mel-spectrograms for such audio often surpass the models' capacity, leading to outputs that are blurred or lack coherence. In this paper, we begin by investigating the critical role of U-Net in Mel-spectrogram generation. Our analysis shows that in U-Net structure, high-frequency components in skip-connections and the backbone influence texture and detail, while low-frequency components in the backbone are critical for the diffusion denoising process. We further propose ``Mel-Refine'', a plug-and-play approach that enhances Mel-spectrogram texture and detail by adjusting different component weights during inference. Our method requires no additional training or fine-tuning and is fully compatible with any diffusion-based TTA architecture. Experimental results show that our approach boosts performance metrics of the latest TTA model Tango2 by 25\%, demonstrating its effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08577 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mel-Refine: A Plug-and-Play Approach to Refine Mel-Spectrogram in Audio Generation Guo, Hongming Fu, Ruibo Geng, Yizhong Liu, Shuai Shi, Shuchen Wang, Tao Qiang, Chunyu Li, Chenxing Li, Ya Wen, Zhengqi Liu, Yukun Liu, Xuefei Sound Multimedia Audio and Speech Processing Text-to-audio (TTA) model is capable of generating diverse audio from textual prompts. However, most mainstream TTA models, which predominantly rely on Mel-spectrograms, still face challenges in producing audio with rich content. The intricate details and texture required in Mel-spectrograms for such audio often surpass the models' capacity, leading to outputs that are blurred or lack coherence. In this paper, we begin by investigating the critical role of U-Net in Mel-spectrogram generation. Our analysis shows that in U-Net structure, high-frequency components in skip-connections and the backbone influence texture and detail, while low-frequency components in the backbone are critical for the diffusion denoising process. We further propose ``Mel-Refine'', a plug-and-play approach that enhances Mel-spectrogram texture and detail by adjusting different component weights during inference. Our method requires no additional training or fine-tuning and is fully compatible with any diffusion-based TTA architecture. Experimental results show that our approach boosts performance metrics of the latest TTA model Tango2 by 25\%, demonstrating its effectiveness. |
| title | Mel-Refine: A Plug-and-Play Approach to Refine Mel-Spectrogram in Audio Generation |
| topic | Sound Multimedia Audio and Speech Processing |
| url | https://arxiv.org/abs/2412.08577 |