Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918473068707840 |
|---|---|
| author | Liu, Carson Yu Mohammadi, Gelareh Song, Yang Johal, Wafa |
| author_facet | Liu, Carson Yu Mohammadi, Gelareh Song, Yang Johal, Wafa |
| contents | Embodied agents, in the form of virtual agents or social robots, are rapidly becoming more widespread. In human-human interactions, humans use nonverbal behaviours to convey their attitudes, feelings, and intentions. Therefore, this capability is also required for embodied agents in order to enhance the quality and effectiveness of their interactions with humans. In this paper, we propose a novel framework that can generate sequences of joint angles from the speech text and speech audio utterances. Based on a conditional Generative Adversarial Network (GAN), our proposed neural network model learns the relationships between the co-speech gestures and both semantic and acoustic features from the speech input. In order to train our neural network model, we employ a public dataset containing co-speech gestures with corresponding speech audio utterances, which were captured from a single male native English speaker. The results from both objective and subjective evaluations demonstrate the efficacy of our gesture-generation framework for Robots and Embodied Agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_09346 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents Liu, Carson Yu Mohammadi, Gelareh Song, Yang Johal, Wafa Artificial Intelligence Robotics I.2; I.2.9 Embodied agents, in the form of virtual agents or social robots, are rapidly becoming more widespread. In human-human interactions, humans use nonverbal behaviours to convey their attitudes, feelings, and intentions. Therefore, this capability is also required for embodied agents in order to enhance the quality and effectiveness of their interactions with humans. In this paper, we propose a novel framework that can generate sequences of joint angles from the speech text and speech audio utterances. Based on a conditional Generative Adversarial Network (GAN), our proposed neural network model learns the relationships between the co-speech gestures and both semantic and acoustic features from the speech input. In order to train our neural network model, we employ a public dataset containing co-speech gestures with corresponding speech audio utterances, which were captured from a single male native English speaker. The results from both objective and subjective evaluations demonstrate the efficacy of our gesture-generation framework for Robots and Embodied Agents. |
| title | Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents |
| topic | Artificial Intelligence Robotics I.2; I.2.9 |
| url | https://arxiv.org/abs/2309.09346 |