OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918131574767616 |
|---|---|
| author | Park, Jeongkyun Hwang, Jung-Wook Choi, Kwanghee Lee, Seung-Hyun Ahn, Jun Hwan Park, Rae-Hong Park, Hyung-Min |
| author_facet | Park, Jeongkyun Hwang, Jung-Wook Choi, Kwanghee Lee, Seung-Hyun Ahn, Jun Hwan Park, Rae-Hong Park, Hyung-Min |
| contents | Inspired by humans comprehending speech in a multi-modal manner, various audio-visual datasets have been constructed. However, most existing datasets focus on English, induce dependencies with various prediction models during dataset preparation, and have only a small number of multi-view videos. To mitigate the limitations, we recently developed the Open Large-scale Korean Audio-Visual Speech (OLKAVS) dataset, which is the largest among publicly available audio-visual speech datasets. The dataset contains 1,150 hours of transcribed audio from 1,107 Korean speakers in a studio setup with nine different viewpoints and various noise situations. We also provide the pre-trained baseline models for two tasks, audio-visual speech recognition and lip reading. We conducted experiments based on the models to verify the effectiveness of multi-modal and multi-view training over uni-modal and frontal-view-only training. We expect the OLKAVS dataset to facilitate multi-modal research in broader areas such as Korean speech recognition, speaker recognition, pronunciation level classification, and mouth motion analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2301_06375 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset Park, Jeongkyun Hwang, Jung-Wook Choi, Kwanghee Lee, Seung-Hyun Ahn, Jun Hwan Park, Rae-Hong Park, Hyung-Min Multimedia Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning Sound Inspired by humans comprehending speech in a multi-modal manner, various audio-visual datasets have been constructed. However, most existing datasets focus on English, induce dependencies with various prediction models during dataset preparation, and have only a small number of multi-view videos. To mitigate the limitations, we recently developed the Open Large-scale Korean Audio-Visual Speech (OLKAVS) dataset, which is the largest among publicly available audio-visual speech datasets. The dataset contains 1,150 hours of transcribed audio from 1,107 Korean speakers in a studio setup with nine different viewpoints and various noise situations. We also provide the pre-trained baseline models for two tasks, audio-visual speech recognition and lip reading. We conducted experiments based on the models to verify the effectiveness of multi-modal and multi-view training over uni-modal and frontal-view-only training. We expect the OLKAVS dataset to facilitate multi-modal research in broader areas such as Korean speech recognition, speaker recognition, pronunciation level classification, and mouth motion analysis. |
| title | OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset |
| topic | Multimedia Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning Sound |
| url | https://arxiv.org/abs/2301.06375 |