Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2403.09527 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909197298302976 |
|---|---|
| author | Liang, Jinhua Zhang, Huan Liu, Haohe Cao, Yin Kong, Qiuqiang Liu, Xubo Wang, Wenwu Plumbley, Mark D. Phan, Huy Benetos, Emmanouil |
| author_facet | Liang, Jinhua Zhang, Huan Liu, Haohe Cao, Yin Kong, Qiuqiang Liu, Xubo Wang, Wenwu Plumbley, Mark D. Phan, Huy Benetos, Emmanouil |
| contents | We introduce WavCraft, a collective system that leverages large language models (LLMs) to connect diverse task-specific models for audio content creation and editing. Specifically, WavCraft describes the content of raw audio materials in natural language and prompts the LLM conditioned on audio descriptions and user requests. WavCraft leverages the in-context learning ability of the LLM to decomposes users' instructions into several tasks and tackle each task collaboratively with the particular module. Through task decomposition along with a set of task-specific models, WavCraft follows the input instruction to create or edit audio content with more details and rationales, facilitating user control. In addition, WavCraft is able to cooperate with users via dialogue interaction and even produce the audio content without explicit user commands. Experiments demonstrate that WavCraft yields a better performance than existing methods, especially when adjusting the local regions of audio clips. Moreover, WavCraft can follow complex instructions to edit and create audio content on the top of input recordings, facilitating audio producers in a broader range of applications. Our implementation and demos are available at this https://github.com/JinhuaLiang/WavCraft. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_09527 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WavCraft: Audio Editing and Generation with Large Language Models Liang, Jinhua Zhang, Huan Liu, Haohe Cao, Yin Kong, Qiuqiang Liu, Xubo Wang, Wenwu Plumbley, Mark D. Phan, Huy Benetos, Emmanouil Audio and Speech Processing We introduce WavCraft, a collective system that leverages large language models (LLMs) to connect diverse task-specific models for audio content creation and editing. Specifically, WavCraft describes the content of raw audio materials in natural language and prompts the LLM conditioned on audio descriptions and user requests. WavCraft leverages the in-context learning ability of the LLM to decomposes users' instructions into several tasks and tackle each task collaboratively with the particular module. Through task decomposition along with a set of task-specific models, WavCraft follows the input instruction to create or edit audio content with more details and rationales, facilitating user control. In addition, WavCraft is able to cooperate with users via dialogue interaction and even produce the audio content without explicit user commands. Experiments demonstrate that WavCraft yields a better performance than existing methods, especially when adjusting the local regions of audio clips. Moreover, WavCraft can follow complex instructions to edit and create audio content on the top of input recordings, facilitating audio producers in a broader range of applications. Our implementation and demos are available at this https://github.com/JinhuaLiang/WavCraft. |
| title | WavCraft: Audio Editing and Generation with Large Language Models |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2403.09527 |