Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2407.08101 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911593244131328 |
|---|---|
| author | Panchal, Sunny Bhattacharyya, Apratim Berger, Guillaume Mercier, Antoine Bohm, Cornelius Dietrichkeit, Florian Pourreza, Reza Li, Xuanlin Madan, Pulkit Lee, Mingu Todorovich, Mark Bax, Ingo Memisevic, Roland |
| author_facet | Panchal, Sunny Bhattacharyya, Apratim Berger, Guillaume Mercier, Antoine Bohm, Cornelius Dietrichkeit, Florian Pourreza, Reza Li, Xuanlin Madan, Pulkit Lee, Mingu Todorovich, Mark Bax, Ingo Memisevic, Roland |
| contents | Vision-language models have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions, where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time, are an open challenge. In this work, we present the QEVD benchmark and dataset, which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching -- a task which intrinsically requires monitoring live user activity and providing immediate feedback. The benchmark requires vision-language models to recognize complex human actions, identify possible mistakes, and provide appropriate feedback in real-time. Our experiments reveal the limitations of existing state-of-the-art vision-language models for such asynchronous situated interactions. Motivated by this, we propose a simple end-to-end streaming baseline that can respond asynchronously to human actions with appropriate feedback at the appropriate time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_08101 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction Panchal, Sunny Bhattacharyya, Apratim Berger, Guillaume Mercier, Antoine Bohm, Cornelius Dietrichkeit, Florian Pourreza, Reza Li, Xuanlin Madan, Pulkit Lee, Mingu Todorovich, Mark Bax, Ingo Memisevic, Roland Computer Vision and Pattern Recognition Vision-language models have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions, where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time, are an open challenge. In this work, we present the QEVD benchmark and dataset, which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching -- a task which intrinsically requires monitoring live user activity and providing immediate feedback. The benchmark requires vision-language models to recognize complex human actions, identify possible mistakes, and provide appropriate feedback in real-time. Our experiments reveal the limitations of existing state-of-the-art vision-language models for such asynchronous situated interactions. Motivated by this, we propose a simple end-to-end streaming baseline that can respond asynchronously to human actions with appropriate feedback at the appropriate time. |
| title | What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.08101 |