VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913170786877440 |
|---|---|
| author | Mazumdar, Amrita Park, Seonwook Roy, Rajarshi Srihari, Nikhil Wang, Shengze Zhou, Yuhao Wang, Julia Nagano, Koki De Mello, Shalini |
| author_facet | Mazumdar, Amrita Park, Seonwook Roy, Rajarshi Srihari, Nikhil Wang, Shengze Zhou, Yuhao Wang, Julia Nagano, Koki De Mello, Shalini |
| contents | Natural human conversation is full-duplex and audio-visual: people simultaneously speak and listen while continuously interpreting and producing nonverbal cues, such as nods, smiles, and gestures. To support successful human-agent interaction, agents must model full-duplex audiovisual conversation; however, existing full-duplex benchmarks evaluate only speech. In this work, we present VideoFDB, the first benchmark to evaluate full-duplex audio-visual-to-audio-visual (AV2AV) conversational agents. VideoFDB contributes (i) 237 dyadic clips spanning 11 nonverbal conversational dynamics from real-world video calls, (ii) a taxonomy separating perception from generation behaviors, and (iii) a rubric-based LM-as-judge evaluation framework with interpretable axes for assessing conversational quality with respect to nonverbal conversational dynamics. Across open- and closed-source vision-speech agents, we find systematic failure modes: captioning collapse and visual-stream ignorance, and we show that current systems exploit vision for explicit visual question answering but not for the streaming joint audiovisual grounding required in natural conversation. We further evaluate cascaded speech-to-avatar systems and find that their architecture fundamentally precludes the production of full-duplex nonverbal cues. As the first benchmark for full-duplex AV2AV interaction, VideoFDB establishes a foundation for systematic evaluation and, we hope, will accelerate the advancement and development of next-generation multimodal conversational agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_30256 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents Mazumdar, Amrita Park, Seonwook Roy, Rajarshi Srihari, Nikhil Wang, Shengze Zhou, Yuhao Wang, Julia Nagano, Koki De Mello, Shalini Computer Vision and Pattern Recognition Computation and Language Human-Computer Interaction Natural human conversation is full-duplex and audio-visual: people simultaneously speak and listen while continuously interpreting and producing nonverbal cues, such as nods, smiles, and gestures. To support successful human-agent interaction, agents must model full-duplex audiovisual conversation; however, existing full-duplex benchmarks evaluate only speech. In this work, we present VideoFDB, the first benchmark to evaluate full-duplex audio-visual-to-audio-visual (AV2AV) conversational agents. VideoFDB contributes (i) 237 dyadic clips spanning 11 nonverbal conversational dynamics from real-world video calls, (ii) a taxonomy separating perception from generation behaviors, and (iii) a rubric-based LM-as-judge evaluation framework with interpretable axes for assessing conversational quality with respect to nonverbal conversational dynamics. Across open- and closed-source vision-speech agents, we find systematic failure modes: captioning collapse and visual-stream ignorance, and we show that current systems exploit vision for explicit visual question answering but not for the streaming joint audiovisual grounding required in natural conversation. We further evaluate cascaded speech-to-avatar systems and find that their architecture fundamentally precludes the production of full-duplex nonverbal cues. As the first benchmark for full-duplex AV2AV interaction, VideoFDB establishes a foundation for systematic evaluation and, we hope, will accelerate the advancement and development of next-generation multimodal conversational agents. |
| title | VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents |
| topic | Computer Vision and Pattern Recognition Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2605.30256 |