Towards Open-ended Visual Quality Comparison
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911788430262272 |
|---|---|
| author | Wu, Haoning Zhu, Hanwei Zhang, Zicheng Zhang, Erli Chen, Chaofeng Liao, Liang Li, Chunyi Wang, Annan Sun, Wenxiu Yan, Qiong Liu, Xiaohong Zhai, Guangtao Wang, Shiqi Lin, Weisi |
| author_facet | Wu, Haoning Zhu, Hanwei Zhang, Zicheng Zhang, Erli Chen, Chaofeng Liao, Liang Li, Chunyi Wang, Annan Sun, Wenxiu Yan, Qiong Liu, Xiaohong Zhai, Guangtao Wang, Shiqi Lin, Weisi |
| contents | Comparative settings (e.g. pairwise choice, listwise ranking) have been adopted by a wide range of subjective studies for image quality assessment (IQA), as it inherently standardizes the evaluation criteria across different observers and offer more clear-cut responses. In this work, we extend the edge of emerging large multi-modality models (LMMs) to further advance visual quality comparison into open-ended settings, that 1) can respond to open-range questions on quality comparison; 2) can provide detailed reasonings beyond direct answers. To this end, we propose the Co-Instruct. To train this first-of-its-kind open-source open-ended visual quality comparer, we collect the Co-Instruct-562K dataset, from two sources: (a) LLM-merged single image quality description, (b) GPT-4V "teacher" responses on unlabeled data. Furthermore, to better evaluate this setting, we propose the MICBench, the first benchmark on multi-image comparison for LMMs. We demonstrate that Co-Instruct not only achieves in average 30% higher accuracy than state-of-the-art open-source LMMs, but also outperforms GPT-4V (its teacher), on both existing related benchmarks and the proposed MICBench. Our model is published at https://huggingface.co/q-future/co-instruct. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16641 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Open-ended Visual Quality Comparison Wu, Haoning Zhu, Hanwei Zhang, Zicheng Zhang, Erli Chen, Chaofeng Liao, Liang Li, Chunyi Wang, Annan Sun, Wenxiu Yan, Qiong Liu, Xiaohong Zhai, Guangtao Wang, Shiqi Lin, Weisi Computer Vision and Pattern Recognition Comparative settings (e.g. pairwise choice, listwise ranking) have been adopted by a wide range of subjective studies for image quality assessment (IQA), as it inherently standardizes the evaluation criteria across different observers and offer more clear-cut responses. In this work, we extend the edge of emerging large multi-modality models (LMMs) to further advance visual quality comparison into open-ended settings, that 1) can respond to open-range questions on quality comparison; 2) can provide detailed reasonings beyond direct answers. To this end, we propose the Co-Instruct. To train this first-of-its-kind open-source open-ended visual quality comparer, we collect the Co-Instruct-562K dataset, from two sources: (a) LLM-merged single image quality description, (b) GPT-4V "teacher" responses on unlabeled data. Furthermore, to better evaluate this setting, we propose the MICBench, the first benchmark on multi-image comparison for LMMs. We demonstrate that Co-Instruct not only achieves in average 30% higher accuracy than state-of-the-art open-source LMMs, but also outperforms GPT-4V (its teacher), on both existing related benchmarks and the proposed MICBench. Our model is published at https://huggingface.co/q-future/co-instruct. |
| title | Towards Open-ended Visual Quality Comparison |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.16641 |