Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913221132156928 |
|---|---|
| author | Kim, Jaeyeon Hwang, Injune Lee, Kyogu |
| author_facet | Kim, Jaeyeon Hwang, Injune Lee, Kyogu |
| contents | We propose a framework to learn semantics from raw audio signals using two types of representations, encoding contextual and phonetic information respectively. Specifically, we introduce a speech-to-unit processing pipeline that captures two types of representations with different time resolutions. For the language model, we adopt a dual-channel architecture to incorporate both types of representation. We also present new training objectives, masked context reconstruction and masked context prediction, that push models to learn semantics effectively. Experiments on the sSIMI metric of Zero Resource Speech Benchmark 2021 and Fluent Speech Command dataset show our framework learns semantics better than models trained with only one type of representation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01298 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations Kim, Jaeyeon Hwang, Injune Lee, Kyogu Audio and Speech Processing Artificial Intelligence Sound We propose a framework to learn semantics from raw audio signals using two types of representations, encoding contextual and phonetic information respectively. Specifically, we introduce a speech-to-unit processing pipeline that captures two types of representations with different time resolutions. For the language model, we adopt a dual-channel architecture to incorporate both types of representation. We also present new training objectives, masked context reconstruction and masked context prediction, that push models to learn semantics effectively. Experiments on the sSIMI metric of Zero Resource Speech Benchmark 2021 and Fluent Speech Command dataset show our framework learns semantics better than models trained with only one type of representation. |
| title | Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations |
| topic | Audio and Speech Processing Artificial Intelligence Sound |
| url | https://arxiv.org/abs/2402.01298 |