Visual Robustness Benchmark for Visual Question Answering (VQA)
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909369872941056 |
|---|---|
| author | Ishmam, Md Farhan Tashdeed, Ishmam Saadat, Talukder Asir Ashmafee, Md Hamjajul Kamal, Abu Raihan Mostofa Hossain, Md. Azam |
| author_facet | Ishmam, Md Farhan Tashdeed, Ishmam Saadat, Talukder Asir Ashmafee, Md Hamjajul Kamal, Abu Raihan Mostofa Hossain, Md. Azam |
| contents | Can Visual Question Answering (VQA) systems perform just as well when deployed in the real world? Or are they susceptible to realistic corruption effects e.g. image blur, which can be detrimental in sensitive applications, such as medical VQA? While linguistic or textual robustness has been thoroughly explored in the VQA literature, there has yet to be any significant work on the visual robustness of VQA models. We propose the first large-scale benchmark comprising 213,000 augmented images, challenging the visual robustness of multiple VQA models and assessing the strength of realistic visual corruptions. Additionally, we have designed several robustness evaluation metrics that can be aggregated into a unified metric and tailored to fit a variety of use cases. Our experiments reveal several insights into the relationships between model size, performance, and robustness with the visual corruptions. Our benchmark highlights the need for a balanced approach in model development that considers model performance without compromising the robustness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03386 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Visual Robustness Benchmark for Visual Question Answering (VQA) Ishmam, Md Farhan Tashdeed, Ishmam Saadat, Talukder Asir Ashmafee, Md Hamjajul Kamal, Abu Raihan Mostofa Hossain, Md. Azam Computer Vision and Pattern Recognition Can Visual Question Answering (VQA) systems perform just as well when deployed in the real world? Or are they susceptible to realistic corruption effects e.g. image blur, which can be detrimental in sensitive applications, such as medical VQA? While linguistic or textual robustness has been thoroughly explored in the VQA literature, there has yet to be any significant work on the visual robustness of VQA models. We propose the first large-scale benchmark comprising 213,000 augmented images, challenging the visual robustness of multiple VQA models and assessing the strength of realistic visual corruptions. Additionally, we have designed several robustness evaluation metrics that can be aggregated into a unified metric and tailored to fit a variety of use cases. Our experiments reveal several insights into the relationships between model size, performance, and robustness with the visual corruptions. Our benchmark highlights the need for a balanced approach in model development that considers model performance without compromising the robustness. |
| title | Visual Robustness Benchmark for Visual Question Answering (VQA) |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.03386 |