Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.12831 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911157403516928 |
|---|---|
| author | Amir, Javeria Attaria, Farwa Jabeen, Mah Noor, Umara Rashid, Zahid |
| author_facet | Amir, Javeria Attaria, Farwa Jabeen, Mah Noor, Umara Rashid, Zahid |
| contents | Recent developments in voice cloning and talking head generation demonstrate impressive capabilities in synthesizing natural speech and realistic lip synchronization. Current methods typically require and are trained on large scale datasets and computationally intensive processes using clean studio recorded inputs that is infeasible in noisy or low resource environments. In this paper, we introduce a new modular pipeline comprising Tortoise text to speech. It is a transformer based latent diffusion model that can perform high fidelity zero shot voice cloning given only a few training samples. We use a lightweight generative adversarial network architecture for robust real time lip synchronization. The solution will contribute to many essential tasks concerning less reliance on massive pre training generation of emotionally expressive speech and lip synchronization in noisy and unconstrained scenarios. The modular structure of the pipeline allows an easy extension for future multi modal and text guided voice modulation and it could be used in real world systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12831 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Lightweight Pipeline for Noisy Speech Voice Cloning and Accurate Lip Sync Synthesis Amir, Javeria Attaria, Farwa Jabeen, Mah Noor, Umara Rashid, Zahid Sound Artificial Intelligence Recent developments in voice cloning and talking head generation demonstrate impressive capabilities in synthesizing natural speech and realistic lip synchronization. Current methods typically require and are trained on large scale datasets and computationally intensive processes using clean studio recorded inputs that is infeasible in noisy or low resource environments. In this paper, we introduce a new modular pipeline comprising Tortoise text to speech. It is a transformer based latent diffusion model that can perform high fidelity zero shot voice cloning given only a few training samples. We use a lightweight generative adversarial network architecture for robust real time lip synchronization. The solution will contribute to many essential tasks concerning less reliance on massive pre training generation of emotionally expressive speech and lip synchronization in noisy and unconstrained scenarios. The modular structure of the pipeline allows an easy extension for future multi modal and text guided voice modulation and it could be used in real world systems. |
| title | A Lightweight Pipeline for Noisy Speech Voice Cloning and Accurate Lip Sync Synthesis |
| topic | Sound Artificial Intelligence |
| url | https://arxiv.org/abs/2509.12831 |