Long-Form Answers to Visual Questions from Blind and Low Vision People
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915408678748160 |
|---|---|
| author | Huh, Mina Xu, Fangyuan Peng, Yi-Hao Chen, Chongyan Murugu, Hansika Gurari, Danna Choi, Eunsol Pavel, Amy |
| author_facet | Huh, Mina Xu, Fangyuan Peng, Yi-Hao Chen, Chongyan Murugu, Hansika Gurari, Danna Choi, Eunsol Pavel, Amy |
| contents | Vision language models can now generate long-form answers to questions about images - long-form visual question answers (LFVQA). We contribute VizWiz-LF, a dataset of long-form answers to visual questions posed by blind and low vision (BLV) users. VizWiz-LF contains 4.2k long-form answers to 600 visual questions, collected from human expert describers and six VQA models. We develop and annotate functional roles of sentences of LFVQA and demonstrate that long-form answers contain information beyond the question answer such as explanations and suggestions. We further conduct automatic and human evaluations with BLV and sighted people to evaluate long-form answers. BLV people perceive both human-written and generated long-form answers to be plausible, but generated answers often hallucinate incorrect visual details, especially for unanswerable visual questions (e.g., blurry or irrelevant images). To reduce hallucinations, we evaluate the ability of VQA models to abstain from answering unanswerable questions across multiple prompting strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_06303 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Long-Form Answers to Visual Questions from Blind and Low Vision People Huh, Mina Xu, Fangyuan Peng, Yi-Hao Chen, Chongyan Murugu, Hansika Gurari, Danna Choi, Eunsol Pavel, Amy Computation and Language Computer Vision and Pattern Recognition Vision language models can now generate long-form answers to questions about images - long-form visual question answers (LFVQA). We contribute VizWiz-LF, a dataset of long-form answers to visual questions posed by blind and low vision (BLV) users. VizWiz-LF contains 4.2k long-form answers to 600 visual questions, collected from human expert describers and six VQA models. We develop and annotate functional roles of sentences of LFVQA and demonstrate that long-form answers contain information beyond the question answer such as explanations and suggestions. We further conduct automatic and human evaluations with BLV and sighted people to evaluate long-form answers. BLV people perceive both human-written and generated long-form answers to be plausible, but generated answers often hallucinate incorrect visual details, especially for unanswerable visual questions (e.g., blurry or irrelevant images). To reduce hallucinations, we evaluate the ability of VQA models to abstain from answering unanswerable questions across multiple prompting strategies. |
| title | Long-Form Answers to Visual Questions from Blind and Low Vision People |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.06303 |