Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916662601580544 |
|---|---|
| author | Lu, Fan Wu, Wei Zheng, Kecheng Ma, Shuailei Gong, Biao Liu, Jiawei Zhai, Wei Cao, Yang Shen, Yujun Zha, Zheng-Jun |
| author_facet | Lu, Fan Wu, Wei Zheng, Kecheng Ma, Shuailei Gong, Biao Liu, Jiawei Zhai, Wei Cao, Yang Shen, Yujun Zha, Zheng-Jun |
| contents | Generating detailed captions comprehending text-rich visual content in images has received growing attention for Large Vision-Language Models (LVLMs). However, few studies have developed benchmarks specifically tailored for detailed captions to measure their accuracy and comprehensiveness. In this paper, we introduce a detailed caption benchmark, termed as CompreCap, to evaluate the visual context from a directed scene graph view. Concretely, we first manually segment the image into semantically meaningful regions (i.e., semantic segmentation mask) according to common-object vocabulary, while also distinguishing attributes of objects within all those regions. Then directional relation labels of these objects are annotated to compose a directed scene graph that can well encode rich compositional information of the image. Based on our directed scene graph, we develop a pipeline to assess the generated detailed captions from LVLMs on multiple levels, including the object-level coverage, the accuracy of attribute descriptions, the score of key relationships, etc. Experimental results on the CompreCap dataset confirm that our evaluation method aligns closely with human evaluation scores across LVLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08614 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning Lu, Fan Wu, Wei Zheng, Kecheng Ma, Shuailei Gong, Biao Liu, Jiawei Zhai, Wei Cao, Yang Shen, Yujun Zha, Zheng-Jun Computer Vision and Pattern Recognition Generating detailed captions comprehending text-rich visual content in images has received growing attention for Large Vision-Language Models (LVLMs). However, few studies have developed benchmarks specifically tailored for detailed captions to measure their accuracy and comprehensiveness. In this paper, we introduce a detailed caption benchmark, termed as CompreCap, to evaluate the visual context from a directed scene graph view. Concretely, we first manually segment the image into semantically meaningful regions (i.e., semantic segmentation mask) according to common-object vocabulary, while also distinguishing attributes of objects within all those regions. Then directional relation labels of these objects are annotated to compose a directed scene graph that can well encode rich compositional information of the image. Based on our directed scene graph, we develop a pipeline to assess the generated detailed captions from LVLMs on multiple levels, including the object-level coverage, the accuracy of attribute descriptions, the score of key relationships, etc. Experimental results on the CompreCap dataset confirm that our evaluation method aligns closely with human evaluation scores across LVLMs. |
| title | Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.08614 |