Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.12026 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916587199528960 |
|---|---|
| author | Dwibedi, Debidatta Jain, Vidhi Tompson, Jonathan Zisserman, Andrew Aytar, Yusuf |
| author_facet | Dwibedi, Debidatta Jain, Vidhi Tompson, Jonathan Zisserman, Andrew Aytar, Yusuf |
| contents | We introduce FlexCap, a vision-language model that generates region-specific descriptions of varying lengths. FlexCap is trained to produce length-conditioned captions for input boxes, enabling control over information density, with descriptions ranging from concise object labels to detailed captions. To achieve this, we create large-scale training datasets of image region descriptions with varying lengths from captioned web images. We demonstrate FlexCap's effectiveness in several applications: first, it achieves strong performance in dense captioning tasks on the Visual Genome dataset. Second, we show how FlexCap's localized descriptions can serve as input to a large language model to create a visual question answering (VQA) system, achieving state-of-the-art zero-shot performance on multiple VQA benchmarks. Our experiments illustrate FlexCap's utility for tasks including image labeling, object attribute recognition, and visual dialog. Project webpage: https://flex-cap.github.io . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12026 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FlexCap: Describe Anything in Images in Controllable Detail Dwibedi, Debidatta Jain, Vidhi Tompson, Jonathan Zisserman, Andrew Aytar, Yusuf Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning We introduce FlexCap, a vision-language model that generates region-specific descriptions of varying lengths. FlexCap is trained to produce length-conditioned captions for input boxes, enabling control over information density, with descriptions ranging from concise object labels to detailed captions. To achieve this, we create large-scale training datasets of image region descriptions with varying lengths from captioned web images. We demonstrate FlexCap's effectiveness in several applications: first, it achieves strong performance in dense captioning tasks on the Visual Genome dataset. Second, we show how FlexCap's localized descriptions can serve as input to a large language model to create a visual question answering (VQA) system, achieving state-of-the-art zero-shot performance on multiple VQA benchmarks. Our experiments illustrate FlexCap's utility for tasks including image labeling, object attribute recognition, and visual dialog. Project webpage: https://flex-cap.github.io . |
| title | FlexCap: Describe Anything in Images in Controllable Detail |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2403.12026 |