Audio Conditioning for Music Generation via Discrete Bottleneck Features
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916339502809088 |
|---|---|
| author | Rouard, Simon Adi, Yossi Copet, Jade Roebel, Axel Défossez, Alexandre |
| author_facet | Rouard, Simon Adi, Yossi Copet, Jade Roebel, Axel Défossez, Alexandre |
| contents | While most music generation models use textual or parametric conditioning (e.g. tempo, harmony, musical genre), we propose to condition a language model based music generation system with audio input. Our exploration involves two distinct strategies. The first strategy, termed textual inversion, leverages a pre-trained text-to-music model to map audio input to corresponding "pseudowords" in the textual embedding space. For the second model we train a music language model from scratch jointly with a text conditioner and a quantized audio feature extractor. At inference time, we can mix textual and audio conditioning and balance them thanks to a novel double classifier free guidance method. We conduct automatic and human studies that validates our approach. We will release the code and we provide music samples on https://musicgenstyle.github.io in order to show the quality of our model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12563 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Audio Conditioning for Music Generation via Discrete Bottleneck Features Rouard, Simon Adi, Yossi Copet, Jade Roebel, Axel Défossez, Alexandre Sound Audio and Speech Processing While most music generation models use textual or parametric conditioning (e.g. tempo, harmony, musical genre), we propose to condition a language model based music generation system with audio input. Our exploration involves two distinct strategies. The first strategy, termed textual inversion, leverages a pre-trained text-to-music model to map audio input to corresponding "pseudowords" in the textual embedding space. For the second model we train a music language model from scratch jointly with a text conditioner and a quantized audio feature extractor. At inference time, we can mix textual and audio conditioning and balance them thanks to a novel double classifier free guidance method. We conduct automatic and human studies that validates our approach. We will release the code and we provide music samples on https://musicgenstyle.github.io in order to show the quality of our model. |
| title | Audio Conditioning for Music Generation via Discrete Bottleneck Features |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2407.12563 |