VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912154862485504 |
|---|---|
| author | Lim, Hyeonseok Shin, Dongjae Song, Seohyun Won, Inho Kim, Minjun Yuk, Junghun Jang, Haneol Lim, KyungTae |
| author_facet | Lim, Hyeonseok Shin, Dongjae Song, Seohyun Won, Inho Kim, Minjun Yuk, Junghun Jang, Haneol Lim, KyungTae |
| contents | We propose the VLR-Bench, a visual question answering (VQA) benchmark for evaluating vision language models (VLMs) based on retrieval augmented generation (RAG). Unlike existing evaluation datasets for external knowledge-based VQA, the proposed VLR-Bench includes five input passages. This allows testing of the ability to determine which passage is useful for answering a given query, a capability lacking in previous research. In this context, we constructed a dataset of 32,000 automatically generated instruction-following examples, which we denote as VLR-IF. This dataset is specifically designed to enhance the RAG capabilities of VLMs by enabling them to learn how to generate appropriate answers based on input passages. We evaluated the validity of the proposed benchmark and training data and verified its performance using the state-of-the-art Llama3-based VLM, the Llava-Llama-3 model. The proposed VLR-Bench and VLR-IF datasets are publicly available online. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10151 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation Lim, Hyeonseok Shin, Dongjae Song, Seohyun Won, Inho Kim, Minjun Yuk, Junghun Jang, Haneol Lim, KyungTae Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language We propose the VLR-Bench, a visual question answering (VQA) benchmark for evaluating vision language models (VLMs) based on retrieval augmented generation (RAG). Unlike existing evaluation datasets for external knowledge-based VQA, the proposed VLR-Bench includes five input passages. This allows testing of the ability to determine which passage is useful for answering a given query, a capability lacking in previous research. In this context, we constructed a dataset of 32,000 automatically generated instruction-following examples, which we denote as VLR-IF. This dataset is specifically designed to enhance the RAG capabilities of VLMs by enabling them to learn how to generate appropriate answers based on input passages. We evaluated the validity of the proposed benchmark and training data and verified its performance using the state-of-the-art Llama3-based VLM, the Llava-Llama-3 model. The proposed VLR-Bench and VLR-IF datasets are publicly available online. |
| title | VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2412.10151 |