KAN-Based Fusion of Dual-Domain for Audio-Driven Facial Landmarks Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913493969534976 |
|---|---|
| author | Vo-Thanh, Hoang-Son Nguyen, Quang-Vinh Kim, Soo-Hyung |
| author_facet | Vo-Thanh, Hoang-Son Nguyen, Quang-Vinh Kim, Soo-Hyung |
| contents | Audio-driven talking face generation is a widely researched topic due to its high applicability. Reconstructing a talking face using audio significantly contributes to fields such as education, healthcare, online conversations, virtual assistants, and virtual reality. Early studies often focused solely on changing the mouth movements, which resulted in outcomes with limited practical applications. Recently, researchers have proposed a new approach of constructing the entire face, including face pose, neck, and shoulders. To achieve this, they need to generate through landmarks. However, creating stable landmarks that align well with the audio is a challenge. In this paper, we propose the KFusion of Dual-Domain model, a robust model that generates landmarks from audio. We separate the audio into two distinct domains to learn emotional information and facial context, then use a fusion mechanism based on the KAN model. Our model demonstrates high efficiency compared to recent models. This will lay the groundwork for the development of the audio-driven talking face generation problem in the future. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_05330 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | KAN-Based Fusion of Dual-Domain for Audio-Driven Facial Landmarks Generation Vo-Thanh, Hoang-Son Nguyen, Quang-Vinh Kim, Soo-Hyung Computer Vision and Pattern Recognition Multimedia Audio-driven talking face generation is a widely researched topic due to its high applicability. Reconstructing a talking face using audio significantly contributes to fields such as education, healthcare, online conversations, virtual assistants, and virtual reality. Early studies often focused solely on changing the mouth movements, which resulted in outcomes with limited practical applications. Recently, researchers have proposed a new approach of constructing the entire face, including face pose, neck, and shoulders. To achieve this, they need to generate through landmarks. However, creating stable landmarks that align well with the audio is a challenge. In this paper, we propose the KFusion of Dual-Domain model, a robust model that generates landmarks from audio. We separate the audio into two distinct domains to learn emotional information and facial context, then use a fusion mechanism based on the KAN model. Our model demonstrates high efficiency compared to recent models. This will lay the groundwork for the development of the audio-driven talking face generation problem in the future. |
| title | KAN-Based Fusion of Dual-Domain for Audio-Driven Facial Landmarks Generation |
| topic | Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2409.05330 |