LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929665044643840 |
|---|---|
| author | Hao, Bowen Zhou, Dongliang Li, Xiaojie Zhang, Xingyu Xie, Liang Wu, Jianlong Yin, Erwei |
| author_facet | Hao, Bowen Zhou, Dongliang Li, Xiaojie Zhang, Xingyu Xie, Liang Wu, Jianlong Yin, Erwei |
| contents | Visual speech recognition (VSR), commonly known as lip reading, has garnered significant attention due to its wide-ranging practical applications. The advent of deep learning techniques and advancements in hardware capabilities have significantly enhanced the performance of lip reading models. Despite these advancements, existing datasets predominantly feature stable video recordings with limited variability in lip movements. This limitation results in models that are highly sensitive to variations encountered in real-world scenarios. To address this issue, we propose a novel framework, LipGen, which aims to improve model robustness by leveraging speech-driven synthetic visual data, thereby mitigating the constraints of current datasets. Additionally, we introduce an auxiliary task that incorporates viseme classification alongside attention mechanisms. This approach facilitates the efficient integration of temporal information, directing the model's focus toward the relevant segments of speech, thereby enhancing discriminative capabilities. Our method demonstrates superior performance compared to the current state-of-the-art on the lip reading in the wild (LRW) dataset and exhibits even more pronounced advantages under challenging conditions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_04204 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition Hao, Bowen Zhou, Dongliang Li, Xiaojie Zhang, Xingyu Xie, Liang Wu, Jianlong Yin, Erwei Computer Vision and Pattern Recognition Multimedia Visual speech recognition (VSR), commonly known as lip reading, has garnered significant attention due to its wide-ranging practical applications. The advent of deep learning techniques and advancements in hardware capabilities have significantly enhanced the performance of lip reading models. Despite these advancements, existing datasets predominantly feature stable video recordings with limited variability in lip movements. This limitation results in models that are highly sensitive to variations encountered in real-world scenarios. To address this issue, we propose a novel framework, LipGen, which aims to improve model robustness by leveraging speech-driven synthetic visual data, thereby mitigating the constraints of current datasets. Additionally, we introduce an auxiliary task that incorporates viseme classification alongside attention mechanisms. This approach facilitates the efficient integration of temporal information, directing the model's focus toward the relevant segments of speech, thereby enhancing discriminative capabilities. Our method demonstrates superior performance compared to the current state-of-the-art on the lip reading in the wild (LRW) dataset and exhibits even more pronounced advantages under challenging conditions. |
| title | LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition |
| topic | Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2501.04204 |