Spoken question answering for visual queries
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908384235618304 |
|---|---|
| author | Shabtay, Nimrod Kons, Zvi Dekel, Avihu Aronowitz, Hagai Hoory, Ron Arbelle, Assaf |
| author_facet | Shabtay, Nimrod Kons, Zvi Dekel, Avihu Aronowitz, Hagai Hoory, Ron Arbelle, Assaf |
| contents | Question answering (QA) systems are designed to answer natural language questions. Visual QA (VQA) and Spoken QA (SQA) systems extend the textual QA system to accept visual and spoken input respectively.
This work aims to create a system that enables user interaction through both speech and images. That is achieved through the fusion of text, speech, and image modalities to tackle the task of spoken VQA (SVQA). The resulting multi-modal model has textual, visual, and spoken inputs and can answer spoken questions on images.
Training and evaluating SVQA models requires a dataset for all three modalities, but no such dataset currently exists. We address this problem by synthesizing VQA datasets using two zero-shot TTS models. Our initial findings indicate that a model trained only with synthesized speech nearly reaches the performance of the upper-bounding model trained on textual QAs. In addition, we show that the choice of the TTS model has a minor impact on accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23308 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Spoken question answering for visual queries Shabtay, Nimrod Kons, Zvi Dekel, Avihu Aronowitz, Hagai Hoory, Ron Arbelle, Assaf Audio and Speech Processing Artificial Intelligence Image and Video Processing Question answering (QA) systems are designed to answer natural language questions. Visual QA (VQA) and Spoken QA (SQA) systems extend the textual QA system to accept visual and spoken input respectively. This work aims to create a system that enables user interaction through both speech and images. That is achieved through the fusion of text, speech, and image modalities to tackle the task of spoken VQA (SVQA). The resulting multi-modal model has textual, visual, and spoken inputs and can answer spoken questions on images. Training and evaluating SVQA models requires a dataset for all three modalities, but no such dataset currently exists. We address this problem by synthesizing VQA datasets using two zero-shot TTS models. Our initial findings indicate that a model trained only with synthesized speech nearly reaches the performance of the upper-bounding model trained on textual QAs. In addition, we show that the choice of the TTS model has a minor impact on accuracy. |
| title | Spoken question answering for visual queries |
| topic | Audio and Speech Processing Artificial Intelligence Image and Video Processing |
| url | https://arxiv.org/abs/2505.23308 |