VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912005400559616 |
|---|---|
| author | Maruf, M. Daw, Arka Mehrab, Kazi Sajeed Manogaran, Harish Babu Neog, Abhilash Sawhney, Medha Khurana, Mridul Balhoff, James P. Bakis, Yasin Altintas, Bahadir Thompson, Matthew J. Campolongo, Elizabeth G. Uyeda, Josef C. Lapp, Hilmar Bart, Henry L. Mabee, Paula M. Su, Yu Chao, Wei-Lun Stewart, Charles Berger-Wolf, Tanya Dahdul, Wasila Karpatne, Anuj |
| author_facet | Maruf, M. Daw, Arka Mehrab, Kazi Sajeed Manogaran, Harish Babu Neog, Abhilash Sawhney, Medha Khurana, Mridul Balhoff, James P. Bakis, Yasin Altintas, Bahadir Thompson, Matthew J. Campolongo, Elizabeth G. Uyeda, Josef C. Lapp, Hilmar Bart, Henry L. Mabee, Paula M. Su, Yu Chao, Wei-Lun Stewart, Charles Berger-Wolf, Tanya Dahdul, Wasila Karpatne, Anuj |
| contents | Images are increasingly becoming the currency for documenting biodiversity on the planet, providing novel opportunities for accelerating scientific discoveries in the field of organismal biology, especially with the advent of large vision-language models (VLMs). We ask if pre-trained VLMs can aid scientists in answering a range of biologically relevant questions without any additional fine-tuning. In this paper, we evaluate the effectiveness of 12 state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images from three groups of organisms: fishes, birds, and butterflies, covering five biologically relevant tasks. We also explore the effects of applying prompting techniques and tests for reasoning hallucination on the performance of VLMs, shedding new light on the capabilities of current SOTA VLMs in answering biologically relevant questions using images. The code and datasets for running all the analyses reported in this paper can be found at https://github.com/sammarfy/VLM4Bio. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16176 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images Maruf, M. Daw, Arka Mehrab, Kazi Sajeed Manogaran, Harish Babu Neog, Abhilash Sawhney, Medha Khurana, Mridul Balhoff, James P. Bakis, Yasin Altintas, Bahadir Thompson, Matthew J. Campolongo, Elizabeth G. Uyeda, Josef C. Lapp, Hilmar Bart, Henry L. Mabee, Paula M. Su, Yu Chao, Wei-Lun Stewart, Charles Berger-Wolf, Tanya Dahdul, Wasila Karpatne, Anuj Computer Vision and Pattern Recognition Images are increasingly becoming the currency for documenting biodiversity on the planet, providing novel opportunities for accelerating scientific discoveries in the field of organismal biology, especially with the advent of large vision-language models (VLMs). We ask if pre-trained VLMs can aid scientists in answering a range of biologically relevant questions without any additional fine-tuning. In this paper, we evaluate the effectiveness of 12 state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images from three groups of organisms: fishes, birds, and butterflies, covering five biologically relevant tasks. We also explore the effects of applying prompting techniques and tests for reasoning hallucination on the performance of VLMs, shedding new light on the capabilities of current SOTA VLMs in answering biologically relevant questions using images. The code and datasets for running all the analyses reported in this paper can be found at https://github.com/sammarfy/VLM4Bio. |
| title | VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.16176 |