A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914729394438144 |
|---|---|
| author | Inadumi, Shun Kawano, Seiya Yuguchi, Akishige Kawanishi, Yasutomo Yoshino, Koichiro |
| author_facet | Inadumi, Shun Kawano, Seiya Yuguchi, Akishige Kawanishi, Yasutomo Yoshino, Koichiro |
| contents | Situated conversations, which refer to visual information as visual question answering (VQA), often contain ambiguities caused by reliance on directive information. This problem is exacerbated because some languages, such as Japanese, often omit subjective or objective terms. Such ambiguities in questions are often clarified by the contexts in conversational situations, such as joint attention with a user or user gaze information. In this study, we propose the Gaze-grounded VQA dataset (GazeVQA) that clarifies ambiguous questions using gaze information by focusing on a clarification process complemented by gaze information. We also propose a method that utilizes gaze target estimation results to improve the accuracy of GazeVQA tasks. Our experimental results showed that the proposed method improved the performance in some cases of a VQA system on GazeVQA and identified some typical problems of GazeVQA tasks that need to be improved. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_17545 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions Inadumi, Shun Kawano, Seiya Yuguchi, Akishige Kawanishi, Yasutomo Yoshino, Koichiro Computation and Language Computer Vision and Pattern Recognition Situated conversations, which refer to visual information as visual question answering (VQA), often contain ambiguities caused by reliance on directive information. This problem is exacerbated because some languages, such as Japanese, often omit subjective or objective terms. Such ambiguities in questions are often clarified by the contexts in conversational situations, such as joint attention with a user or user gaze information. In this study, we propose the Gaze-grounded VQA dataset (GazeVQA) that clarifies ambiguous questions using gaze information by focusing on a clarification process complemented by gaze information. We also propose a method that utilizes gaze target estimation results to improve the accuracy of GazeVQA tasks. Our experimental results showed that the proposed method improved the performance in some cases of a VQA system on GazeVQA and identified some typical problems of GazeVQA tasks that need to be improved. |
| title | A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.17545 |