ImageInWords: Unlocking Hyper-Detailed Image Descriptions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916457190785024 |
|---|---|
| author | Garg, Roopal Burns, Andrea Ayan, Burcu Karagol Bitton, Yonatan Montgomery, Ceslee Onoe, Yasumasa Bunner, Andrew Krishna, Ranjay Baldridge, Jason Soricut, Radu |
| author_facet | Garg, Roopal Burns, Andrea Ayan, Burcu Karagol Bitton, Yonatan Montgomery, Ceslee Onoe, Yasumasa Bunner, Andrew Krishna, Ranjay Baldridge, Jason Soricut, Radu |
| contents | Despite the longstanding adage "an image is worth a thousand words," generating accurate hyper-detailed image descriptions remains unsolved. Trained on short web-scraped image text, vision-language models often generate incomplete descriptions with visual inconsistencies. We address this via a novel data-centric approach with ImageInWords (IIW), a carefully designed human-in-the-loop framework for curating hyper-detailed image descriptions. Human evaluations on IIW data show major gains compared to recent datasets (+66%) and GPT4V (+48%) across comprehensiveness, specificity, hallucinations, and more. We also show that fine-tuning with IIW data improves these metrics by +31% against models trained with prior work, even with only 9k samples. Lastly, we evaluate IIW models with text-to-image generation and vision-language reasoning tasks. Our generated descriptions result in the highest fidelity images, and boost compositional reasoning by up to 6% on ARO, SVO-Probes, and Winoground datasets. We release the IIW Eval benchmark with human judgement labels, object and image-level annotations from our framework, and existing image caption datasets enriched via IIW-model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_02793 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ImageInWords: Unlocking Hyper-Detailed Image Descriptions Garg, Roopal Burns, Andrea Ayan, Burcu Karagol Bitton, Yonatan Montgomery, Ceslee Onoe, Yasumasa Bunner, Andrew Krishna, Ranjay Baldridge, Jason Soricut, Radu Computer Vision and Pattern Recognition Computation and Language Despite the longstanding adage "an image is worth a thousand words," generating accurate hyper-detailed image descriptions remains unsolved. Trained on short web-scraped image text, vision-language models often generate incomplete descriptions with visual inconsistencies. We address this via a novel data-centric approach with ImageInWords (IIW), a carefully designed human-in-the-loop framework for curating hyper-detailed image descriptions. Human evaluations on IIW data show major gains compared to recent datasets (+66%) and GPT4V (+48%) across comprehensiveness, specificity, hallucinations, and more. We also show that fine-tuning with IIW data improves these metrics by +31% against models trained with prior work, even with only 9k samples. Lastly, we evaluate IIW models with text-to-image generation and vision-language reasoning tasks. Our generated descriptions result in the highest fidelity images, and boost compositional reasoning by up to 6% on ARO, SVO-Probes, and Winoground datasets. We release the IIW Eval benchmark with human judgement labels, object and image-level annotations from our framework, and existing image caption datasets enriched via IIW-model. |
| title | ImageInWords: Unlocking Hyper-Detailed Image Descriptions |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2405.02793 |