Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913364502904832 |
|---|---|
| author | Li, Zhenyang Guo, Yangyang Wang, Kejie Chen, Xiaolin Nie, Liqiang Kankanhalli, Mohan |
| author_facet | Li, Zhenyang Guo, Yangyang Wang, Kejie Chen, Xiaolin Nie, Liqiang Kankanhalli, Mohan |
| contents | Visual Commonsense Reasoning (VCR) calls for explanatory reasoning behind question answering over visual scenes. To achieve this goal, a model is required to provide an acceptable rationale as the reason for the predicted answers. Progress on the benchmark dataset stems largely from the recent advancement of Vision-Language Transformers (VL Transformers). These models are first pre-trained on some generic large-scale vision-text datasets, and then the learned representations are transferred to the downstream VCR task. Despite their attractive performance, this paper posits that the VL Transformers do not exhibit visual commonsense, which is the key to VCR. In particular, our empirical results pinpoint several shortcomings of existing VL Transformers: small gains from pre-training, unexpected language bias, limited model architecture for the two inseparable sub-tasks, and neglect of the important object-tag correlation. With these findings, we tentatively suggest some future directions from the aspect of dataset, evaluation metric, and training tricks. We believe this work could make researchers revisit the intuition and goals of VCR, and thus help tackle the remaining challenges in visual reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_16934 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR Li, Zhenyang Guo, Yangyang Wang, Kejie Chen, Xiaolin Nie, Liqiang Kankanhalli, Mohan Computer Vision and Pattern Recognition Visual Commonsense Reasoning (VCR) calls for explanatory reasoning behind question answering over visual scenes. To achieve this goal, a model is required to provide an acceptable rationale as the reason for the predicted answers. Progress on the benchmark dataset stems largely from the recent advancement of Vision-Language Transformers (VL Transformers). These models are first pre-trained on some generic large-scale vision-text datasets, and then the learned representations are transferred to the downstream VCR task. Despite their attractive performance, this paper posits that the VL Transformers do not exhibit visual commonsense, which is the key to VCR. In particular, our empirical results pinpoint several shortcomings of existing VL Transformers: small gains from pre-training, unexpected language bias, limited model architecture for the two inseparable sub-tasks, and neglect of the important object-tag correlation. With these findings, we tentatively suggest some future directions from the aspect of dataset, evaluation metric, and training tricks. We believe this work could make researchers revisit the intuition and goals of VCR, and thus help tackle the remaining challenges in visual reasoning. |
| title | Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.16934 |