Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.06251 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914830675345408 |
|---|---|
| author | Chien, Chung-Ming Tjandra, Andros Vyas, Apoorv Le, Matt Shi, Bowen Hsu, Wei-Ning |
| author_facet | Chien, Chung-Ming Tjandra, Andros Vyas, Apoorv Le, Matt Shi, Bowen Hsu, Wei-Ning |
| contents | As the scale of generative models continues to grow, efficient reuse and adaptation of pre-trained models have become crucial considerations. In this work, we propose Voicebox Adapter, a novel approach that integrates fine-grained conditions into a pre-trained Voicebox speech generation model using a cross-attention module. To ensure a smooth integration of newly added modules with pre-trained ones, we explore various efficient fine-tuning approaches. Our experiment shows that the LoRA with bias-tuning configuration yields the best performance, enhancing controllability without compromising speech quality. Across three fine-grained conditional generation tasks, we demonstrate the effectiveness and resource efficiency of Voicebox Adapter. Follow-up experiments further highlight the robustness of Voicebox Adapter across diverse data setups. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06251 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning Chien, Chung-Ming Tjandra, Andros Vyas, Apoorv Le, Matt Shi, Bowen Hsu, Wei-Ning Audio and Speech Processing Computation and Language As the scale of generative models continues to grow, efficient reuse and adaptation of pre-trained models have become crucial considerations. In this work, we propose Voicebox Adapter, a novel approach that integrates fine-grained conditions into a pre-trained Voicebox speech generation model using a cross-attention module. To ensure a smooth integration of newly added modules with pre-trained ones, we explore various efficient fine-tuning approaches. Our experiment shows that the LoRA with bias-tuning configuration yields the best performance, enhancing controllability without compromising speech quality. Across three fine-grained conditional generation tasks, we demonstrate the effectiveness and resource efficiency of Voicebox Adapter. Follow-up experiments further highlight the robustness of Voicebox Adapter across diverse data setups. |
| title | Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning |
| topic | Audio and Speech Processing Computation and Language |
| url | https://arxiv.org/abs/2406.06251 |