GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913570095104000 |
|---|---|
| author | Li, Baiqi Lin, Zhiqiu Pathak, Deepak Li, Jiayao Fei, Yixin Wu, Kewen Ling, Tiffany Xia, Xide Zhang, Pengchuan Neubig, Graham Ramanan, Deva |
| author_facet | Li, Baiqi Lin, Zhiqiu Pathak, Deepak Li, Jiayao Fei, Yixin Wu, Kewen Ling, Tiffany Xia, Xide Zhang, Pengchuan Neubig, Graham Ramanan, Deva |
| contents | While text-to-visual models now produce photo-realistic images and videos, they struggle with compositional text prompts involving attributes, relationships, and higher-order reasoning such as logic and comparison. In this work, we conduct an extensive human study on GenAI-Bench to evaluate the performance of leading image and video generation models in various aspects of compositional text-to-visual generation. We also compare automated evaluation metrics against our collected human ratings and find that VQAScore -- a metric measuring the likelihood that a VQA model views an image as accurately depicting the prompt -- significantly outperforms previous metrics such as CLIPScore. In addition, VQAScore can improve generation in a black-box manner (without finetuning) via simply ranking a few (3 to 9) candidate images. Ranking by VQAScore is 2x to 3x more effective than other scoring methods like PickScore, HPSv2, and ImageReward at improving human alignment ratings for DALL-E 3 and Stable Diffusion, especially on compositional prompts that require advanced visio-linguistic reasoning. We release a new GenAI-Rank benchmark with over 40,000 human ratings to evaluate scoring metrics on ranking images generated from the same prompt. Lastly, we discuss promising areas for improvement in VQAScore, such as addressing fine-grained visual details. We will release all human ratings (over 80,000) to facilitate scientific benchmarking of both generative models and automated metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13743 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation Li, Baiqi Lin, Zhiqiu Pathak, Deepak Li, Jiayao Fei, Yixin Wu, Kewen Ling, Tiffany Xia, Xide Zhang, Pengchuan Neubig, Graham Ramanan, Deva Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Multimedia While text-to-visual models now produce photo-realistic images and videos, they struggle with compositional text prompts involving attributes, relationships, and higher-order reasoning such as logic and comparison. In this work, we conduct an extensive human study on GenAI-Bench to evaluate the performance of leading image and video generation models in various aspects of compositional text-to-visual generation. We also compare automated evaluation metrics against our collected human ratings and find that VQAScore -- a metric measuring the likelihood that a VQA model views an image as accurately depicting the prompt -- significantly outperforms previous metrics such as CLIPScore. In addition, VQAScore can improve generation in a black-box manner (without finetuning) via simply ranking a few (3 to 9) candidate images. Ranking by VQAScore is 2x to 3x more effective than other scoring methods like PickScore, HPSv2, and ImageReward at improving human alignment ratings for DALL-E 3 and Stable Diffusion, especially on compositional prompts that require advanced visio-linguistic reasoning. We release a new GenAI-Rank benchmark with over 40,000 human ratings to evaluate scoring metrics on ranking images generated from the same prompt. Lastly, we discuss promising areas for improvement in VQAScore, such as addressing fine-grained visual details. We will release all human ratings (over 80,000) to facilitate scientific benchmarking of both generative models and automated metrics. |
| title | GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Multimedia |
| url | https://arxiv.org/abs/2406.13743 |