Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.00245 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916989950230528 |
|---|---|
| author | Sharma, Rizul Jiang, Tianyu Lee, Seokki Aurisano, Jillian |
| author_facet | Sharma, Rizul Jiang, Tianyu Lee, Seokki Aurisano, Jillian |
| contents | In this short paper, we present work evaluating an AI agent's understanding of spoken conversations about data visualizations in an online meeting scenario. There is growing interest in the development of AI-assistants that support meetings, such as by providing assistance with tasks or summarizing a discussion. The quality of this support depends on a model that understands the conversational dialogue. To evaluate this understanding, we introduce a dual-axis testing framework for diagnosing the AI agent's comprehension of spoken conversations about data. Using this framework, we designed a series of tests to evaluate understanding of a novel corpus of 72 spoken conversational dialogues about data visualizations. We examine diverse pipelines and model architectures, LLM vs VLM, and diverse input formats for visualizations (the chart image, its underlying source code, or a hybrid of both) to see how this affects model performance on our tests. Using our evaluation methods, we found that text-only input modalities achieved the best performance (96%) in understanding discussions of visualizations in online meetings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00245 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Can AI agents understand spoken conversations about data visualizations in online meetings? Sharma, Rizul Jiang, Tianyu Lee, Seokki Aurisano, Jillian Human-Computer Interaction Artificial Intelligence In this short paper, we present work evaluating an AI agent's understanding of spoken conversations about data visualizations in an online meeting scenario. There is growing interest in the development of AI-assistants that support meetings, such as by providing assistance with tasks or summarizing a discussion. The quality of this support depends on a model that understands the conversational dialogue. To evaluate this understanding, we introduce a dual-axis testing framework for diagnosing the AI agent's comprehension of spoken conversations about data. Using this framework, we designed a series of tests to evaluate understanding of a novel corpus of 72 spoken conversational dialogues about data visualizations. We examine diverse pipelines and model architectures, LLM vs VLM, and diverse input formats for visualizations (the chart image, its underlying source code, or a hybrid of both) to see how this affects model performance on our tests. Using our evaluation methods, we found that text-only input modalities achieved the best performance (96%) in understanding discussions of visualizations in online meetings. |
| title | Can AI agents understand spoken conversations about data visualizations in online meetings? |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2510.00245 |