Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.08578 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914835997917184 |
|---|---|
| author | Urbanek, Jack Bordes, Florian Astolfi, Pietro Williamson, Mary Sharma, Vasu Romero-Soriano, Adriana |
| author_facet | Urbanek, Jack Bordes, Florian Astolfi, Pietro Williamson, Mary Sharma, Vasu Romero-Soriano, Adriana |
| contents | Curation methods for massive vision-language datasets trade off between dataset size and quality. However, even the highest quality of available curated captions are far too short to capture the rich visual detail in an image. To show the value of dense and highly-aligned image-text pairs, we collect the Densely Captioned Images (DCI) dataset, containing 7805 natural images human-annotated with mask-aligned descriptions averaging above 1000 words each. With precise and reliable captions associated with specific parts of an image, we can evaluate vision-language models' (VLMs) understanding of image content with a novel task that matches each caption with its corresponding subcrop. As current models are often limited to 77 text tokens, we also introduce a summarized version (sDCI) in which each caption length is limited. We show that modern techniques that make progress on standard benchmarks do not correspond with significant improvement on our sDCI based benchmark. Lastly, we finetune CLIP using sDCI and show significant improvements over the baseline despite a small training set. By releasing the first human annotated dense image captioning dataset, we hope to enable the development of new benchmarks or fine-tuning recipes for the next generation of VLMs to come. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_08578 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions Urbanek, Jack Bordes, Florian Astolfi, Pietro Williamson, Mary Sharma, Vasu Romero-Soriano, Adriana Computer Vision and Pattern Recognition Curation methods for massive vision-language datasets trade off between dataset size and quality. However, even the highest quality of available curated captions are far too short to capture the rich visual detail in an image. To show the value of dense and highly-aligned image-text pairs, we collect the Densely Captioned Images (DCI) dataset, containing 7805 natural images human-annotated with mask-aligned descriptions averaging above 1000 words each. With precise and reliable captions associated with specific parts of an image, we can evaluate vision-language models' (VLMs) understanding of image content with a novel task that matches each caption with its corresponding subcrop. As current models are often limited to 77 text tokens, we also introduce a summarized version (sDCI) in which each caption length is limited. We show that modern techniques that make progress on standard benchmarks do not correspond with significant improvement on our sDCI based benchmark. Lastly, we finetune CLIP using sDCI and show significant improvements over the baseline despite a small training set. By releasing the first human annotated dense image captioning dataset, we hope to enable the development of new benchmarks or fine-tuning recipes for the next generation of VLMs to come. |
| title | A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.08578 |