Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.11476 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916792709939200 |
|---|---|
| author | Baker, Tom Nistal, Javier |
| author_facet | Baker, Tom Nistal, Javier |
| contents | Text-to-audio diffusion models produce high-quality and diverse music but many, if not most, of the SOTA models lack the fine-grained, time-varying controls essential for music production. ControlNet enables attaching external controls to a pre-trained generative model by cloning and fine-tuning its encoder on new conditionings. However, this approach incurs a large memory footprint and restricts users to a fixed set of controls. We propose a lightweight, modular architecture that considerably reduces parameter count while matching ControlNet in audio quality and condition adherence. Our method offers greater flexibility and significantly lower memory usage, enabling more efficient training and deployment of independent controls. We conduct extensive objective and subjective evaluations and provide numerous audio examples on the accompanying website at https://lightlatentcontrol.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_11476 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LiLAC: A Lightweight Latent ControlNet for Musical Audio Generation Baker, Tom Nistal, Javier Sound Machine Learning Audio and Speech Processing Text-to-audio diffusion models produce high-quality and diverse music but many, if not most, of the SOTA models lack the fine-grained, time-varying controls essential for music production. ControlNet enables attaching external controls to a pre-trained generative model by cloning and fine-tuning its encoder on new conditionings. However, this approach incurs a large memory footprint and restricts users to a fixed set of controls. We propose a lightweight, modular architecture that considerably reduces parameter count while matching ControlNet in audio quality and condition adherence. Our method offers greater flexibility and significantly lower memory usage, enabling more efficient training and deployment of independent controls. We conduct extensive objective and subjective evaluations and provide numerous audio examples on the accompanying website at https://lightlatentcontrol.github.io |
| title | LiLAC: A Lightweight Latent ControlNet for Musical Audio Generation |
| topic | Sound Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2506.11476 |