Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914957111590912 |
|---|---|
| author | Zhao, Bowen Dirac, Leo Parker Varshavskaya, Paulina |
| author_facet | Zhao, Bowen Dirac, Leo Parker Varshavskaya, Paulina |
| contents | Large vision-language models (VLMs) have become state-of-the-art for many computer vision tasks, with in-context learning (ICL) as a popular adaptation strategy for new ones. But can VLMs learn novel concepts purely from visual demonstrations, or are they limited to adapting to the output format of ICL examples? We propose a new benchmark we call Spatial Visual Ambiguity Tasks (SVAT) that challenges state-of-the-art VLMs to learn new visuospatial tasks in-context. We find that VLMs fail to do this zero-shot, and sometimes continue to fail after finetuning. However, adding simpler data to the training by curriculum learning leads to improved ICL performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17080 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning? Zhao, Bowen Dirac, Leo Parker Varshavskaya, Paulina Computer Vision and Pattern Recognition Computation and Language Large vision-language models (VLMs) have become state-of-the-art for many computer vision tasks, with in-context learning (ICL) as a popular adaptation strategy for new ones. But can VLMs learn novel concepts purely from visual demonstrations, or are they limited to adapting to the output format of ICL examples? We propose a new benchmark we call Spatial Visual Ambiguity Tasks (SVAT) that challenges state-of-the-art VLMs to learn new visuospatial tasks in-context. We find that VLMs fail to do this zero-shot, and sometimes continue to fail after finetuning. However, adding simpler data to the training by curriculum learning leads to improved ICL performance. |
| title | Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning? |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2409.17080 |