Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2410.08470 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916433510793216 |
|---|---|
| author | Li, Jia Yu, Yangchen Chen, Yin Zhang, Yu Jia, Peng Xu, Yunbo Li, Ziqiang Wang, Meng Hong, Richang |
| author_facet | Li, Jia Yu, Yangchen Chen, Yin Zhang, Yu Jia, Peng Xu, Yunbo Li, Ziqiang Wang, Meng Hong, Richang |
| contents | Engagement estimation plays a crucial role in understanding human social behaviors, attracting increasing research interests in fields such as affective computing and human-computer interaction. In this paper, we propose a Dialogue-Aware Transformer framework (DAT) with Modality-Group Fusion (MGF), which relies solely on audio-visual input and is language-independent, for estimating human engagement in conversations. Specifically, our method employs a modality-group fusion strategy that independently fuses audio and visual features within each modality for each person before inferring the entire audio-visual content. This strategy significantly enhances the model's performance and robustness. Additionally, to better estimate the target participant's engagement levels, the introduced Dialogue-Aware Transformer considers both the participant's behavior and cues from their conversational partners. Our method was rigorously tested in the Multi-Domain Engagement Estimation Challenge held by MultiMediate'24, demonstrating notable improvements in engagement-level regression precision over the baseline model. Notably, our approach achieves a CCC score of 0.76 on the NoXi Base test set and an average CCC of 0.64 across the NoXi Base, NoXi-Add, and MPIIGI test sets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08470 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DAT: Dialogue-Aware Transformer with Modality-Group Fusion for Human Engagement Estimation Li, Jia Yu, Yangchen Chen, Yin Zhang, Yu Jia, Peng Xu, Yunbo Li, Ziqiang Wang, Meng Hong, Richang Human-Computer Interaction Computer Vision and Pattern Recognition Engagement estimation plays a crucial role in understanding human social behaviors, attracting increasing research interests in fields such as affective computing and human-computer interaction. In this paper, we propose a Dialogue-Aware Transformer framework (DAT) with Modality-Group Fusion (MGF), which relies solely on audio-visual input and is language-independent, for estimating human engagement in conversations. Specifically, our method employs a modality-group fusion strategy that independently fuses audio and visual features within each modality for each person before inferring the entire audio-visual content. This strategy significantly enhances the model's performance and robustness. Additionally, to better estimate the target participant's engagement levels, the introduced Dialogue-Aware Transformer considers both the participant's behavior and cues from their conversational partners. Our method was rigorously tested in the Multi-Domain Engagement Estimation Challenge held by MultiMediate'24, demonstrating notable improvements in engagement-level regression precision over the baseline model. Notably, our approach achieves a CCC score of 0.76 on the NoXi Base test set and an average CCC of 0.64 across the NoXi Base, NoXi-Add, and MPIIGI test sets. |
| title | DAT: Dialogue-Aware Transformer with Modality-Group Fusion for Human Engagement Estimation |
| topic | Human-Computer Interaction Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.08470 |