Multimodal Machine Learning Can Predict Videoconference Fluidity and Enjoyment
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915189335523328 |
|---|---|
| author | Chang, Andrew Akkaraju, Viswadruth Cogliano, Ray McFadden Poeppel, David Freeman, Dustin |
| author_facet | Chang, Andrew Akkaraju, Viswadruth Cogliano, Ray McFadden Poeppel, David Freeman, Dustin |
| contents | Videoconferencing is now a frequent mode of communication in both professional and informal settings, yet it often lacks the fluidity and enjoyment of in-person conversation. This study leverages multimodal machine learning to predict moments of negative experience in videoconferencing. We sampled thousands of short clips from the RoomReader corpus, extracting audio embeddings, facial actions, and body motion features to train models for identifying low conversational fluidity, low enjoyment, and classifying conversational events (backchanneling, interruption, or gap). Our best models achieved an ROC-AUC of up to 0.87 on hold-out videoconference sessions, with domain-general audio features proving most critical. This work demonstrates that multimodal audio-video signals can effectively predict high-level subjective conversational outcomes. In addition, this is a contribution to research on videoconferencing user experience by showing that multimodal machine learning can be used to identify rare moments of negative user experience for further study or mitigation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_03190 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multimodal Machine Learning Can Predict Videoconference Fluidity and Enjoyment Chang, Andrew Akkaraju, Viswadruth Cogliano, Ray McFadden Poeppel, David Freeman, Dustin Machine Learning Human-Computer Interaction Audio and Speech Processing Image and Video Processing Videoconferencing is now a frequent mode of communication in both professional and informal settings, yet it often lacks the fluidity and enjoyment of in-person conversation. This study leverages multimodal machine learning to predict moments of negative experience in videoconferencing. We sampled thousands of short clips from the RoomReader corpus, extracting audio embeddings, facial actions, and body motion features to train models for identifying low conversational fluidity, low enjoyment, and classifying conversational events (backchanneling, interruption, or gap). Our best models achieved an ROC-AUC of up to 0.87 on hold-out videoconference sessions, with domain-general audio features proving most critical. This work demonstrates that multimodal audio-video signals can effectively predict high-level subjective conversational outcomes. In addition, this is a contribution to research on videoconferencing user experience by showing that multimodal machine learning can be used to identify rare moments of negative user experience for further study or mitigation. |
| title | Multimodal Machine Learning Can Predict Videoconference Fluidity and Enjoyment |
| topic | Machine Learning Human-Computer Interaction Audio and Speech Processing Image and Video Processing |
| url | https://arxiv.org/abs/2501.03190 |