DOCCI: Descriptions of Connected and Contrasting Images
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917654709665792 |
|---|---|
| author | Onoe, Yasumasa Rane, Sunayana Berger, Zachary Bitton, Yonatan Cho, Jaemin Garg, Roopal Ku, Alexander Parekh, Zarana Pont-Tuset, Jordi Tanzer, Garrett Wang, Su Baldridge, Jason |
| author_facet | Onoe, Yasumasa Rane, Sunayana Berger, Zachary Bitton, Yonatan Cho, Jaemin Garg, Roopal Ku, Alexander Parekh, Zarana Pont-Tuset, Jordi Tanzer, Garrett Wang, Su Baldridge, Jason |
| contents | Vision-language datasets are vital for both text-to-image (T2I) and image-to-text (I2T) research. However, current datasets lack descriptions with fine-grained detail that would allow for richer associations to be learned by models. To fill the gap, we introduce Descriptions of Connected and Contrasting Images (DOCCI), a dataset with long, human-annotated English descriptions for 15k images that were taken, curated and donated by a single researcher intent on capturing key challenges such as spatial relations, counting, text rendering, world knowledge, and more. We instruct human annotators to create comprehensive descriptions for each image; these average 136 words in length and are crafted to clearly distinguish each image from those that are related or similar. Each description is highly compositional and typically encompasses multiple challenges. Through both quantitative and qualitative analyses, we demonstrate that DOCCI serves as an effective training resource for image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or superior results compared to highly-performant larger models like LLaVA-1.5 7B and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for text-to-image generation, highlighting the limitations of current text-to-image models in capturing long descriptions and fine details. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_19753 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DOCCI: Descriptions of Connected and Contrasting Images Onoe, Yasumasa Rane, Sunayana Berger, Zachary Bitton, Yonatan Cho, Jaemin Garg, Roopal Ku, Alexander Parekh, Zarana Pont-Tuset, Jordi Tanzer, Garrett Wang, Su Baldridge, Jason Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Vision-language datasets are vital for both text-to-image (T2I) and image-to-text (I2T) research. However, current datasets lack descriptions with fine-grained detail that would allow for richer associations to be learned by models. To fill the gap, we introduce Descriptions of Connected and Contrasting Images (DOCCI), a dataset with long, human-annotated English descriptions for 15k images that were taken, curated and donated by a single researcher intent on capturing key challenges such as spatial relations, counting, text rendering, world knowledge, and more. We instruct human annotators to create comprehensive descriptions for each image; these average 136 words in length and are crafted to clearly distinguish each image from those that are related or similar. Each description is highly compositional and typically encompasses multiple challenges. Through both quantitative and qualitative analyses, we demonstrate that DOCCI serves as an effective training resource for image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or superior results compared to highly-performant larger models like LLaVA-1.5 7B and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for text-to-image generation, highlighting the limitations of current text-to-image models in capturing long descriptions and fine details. |
| title | DOCCI: Descriptions of Connected and Contrasting Images |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2404.19753 |