Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911165820436480 |
|---|---|
| author | Kapuriya, Janak Shaikh, Anwar Goel, Arnav Hira, Medha Singh, Apoorv Saraf, Jay Sanjana Nauriyal, Vaibhav Anand, Avinash Wang, Zhengkui Shah, Rajiv Ratn |
| author_facet | Kapuriya, Janak Shaikh, Anwar Goel, Arnav Hira, Medha Singh, Apoorv Saraf, Jay Sanjana Nauriyal, Vaibhav Anand, Avinash Wang, Zhengkui Shah, Rajiv Ratn |
| contents | In this study, we introduce Vision-Caption aware Supervised FineTuning (VCASFT), a novel learning paradigm designed to enhance the performance of smaller Vision Language Models(VLMs) on scientific visual question answering(VQA) tasks. VCASFT leverages image captions as zero-shot prompts alongside question-answer pairs and instruction-tunes models to yield significant performance improvements. To comprehensively evaluate VCASFT, we benchmark it on ScienceQA, which consists of questions across diverse languages, subjects, and fields, demonstrating its adaptability and effectiveness in a variety of educational contexts. Additionally, to further demonstrate the effectiveness of this technique on lowresource languages, we developed HiSciVQA, a dataset comprising 2,245 high-quality, hand-annotated Hindi multimodal Q&A pairs. This dataset addresses the critical need for low-resource language Q&A datasets and serves as a foundation for testing VCASFT. Additionally, we introduce a novel LLM-based evaluation scheme to evaluate VLMs on HiSciVQA which offers deeper insights into model effectiveness surpassing traditional n-gram matching accuracy metrics. We are committed to advancing the field by open-sourcing all code files and the HiSciVQA dataset for the research community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_16628 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuning Kapuriya, Janak Shaikh, Anwar Goel, Arnav Hira, Medha Singh, Apoorv Saraf, Jay Sanjana Nauriyal, Vaibhav Anand, Avinash Wang, Zhengkui Shah, Rajiv Ratn Computer Vision and Pattern Recognition In this study, we introduce Vision-Caption aware Supervised FineTuning (VCASFT), a novel learning paradigm designed to enhance the performance of smaller Vision Language Models(VLMs) on scientific visual question answering(VQA) tasks. VCASFT leverages image captions as zero-shot prompts alongside question-answer pairs and instruction-tunes models to yield significant performance improvements. To comprehensively evaluate VCASFT, we benchmark it on ScienceQA, which consists of questions across diverse languages, subjects, and fields, demonstrating its adaptability and effectiveness in a variety of educational contexts. Additionally, to further demonstrate the effectiveness of this technique on lowresource languages, we developed HiSciVQA, a dataset comprising 2,245 high-quality, hand-annotated Hindi multimodal Q&A pairs. This dataset addresses the critical need for low-resource language Q&A datasets and serves as a foundation for testing VCASFT. Additionally, we introduce a novel LLM-based evaluation scheme to evaluate VLMs on HiSciVQA which offers deeper insights into model effectiveness surpassing traditional n-gram matching accuracy metrics. We are committed to advancing the field by open-sourcing all code files and the HiSciVQA dataset for the research community. |
| title | Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.16628 |