Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.06152 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913980330541056 |
|---|---|
| author | Jiang, Kaiyuan Sun, Ruoxi Cao, Ying Xu, Yuqi Zhang, Xinran Guo, Junyan Deng, ChengSheng |
| author_facet | Jiang, Kaiyuan Sun, Ruoxi Cao, Ying Xu, Yuqi Zhang, Xinran Guo, Junyan Deng, ChengSheng |
| contents | We present VISTAR, a user-centric, multi-dimensional benchmark for text-to-image (T2I) evaluation that addresses the limitations of existing metrics. VISTAR introduces a two-tier hybrid paradigm: it employs deterministic, scriptable metrics for physically quantifiable attributes (e.g., text rendering, lighting) and a novel Hierarchical Weighted P/N Questioning (HWPQ) scheme that uses constrained vision-language models to assess abstract semantics (e.g., style fusion, cultural fidelity). Grounded in a Delphi study with 120 experts, we defined seven user roles and nine evaluation angles to construct the benchmark, which comprises 2,845 prompts validated by over 15,000 human pairwise comparisons. Our metrics achieve high human alignment (>75%), with the HWPQ scheme reaching 85.9% accuracy on abstract semantics, significantly outperforming VQA baselines. Comprehensive evaluation of state-of-the-art models reveals no universal champion, as role-weighted scores reorder rankings and provide actionable guidance for domain-specific deployment. All resources are publicly released to foster reproducible T2I assessment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_06152 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VISTAR:A User-Centric and Role-Driven Benchmark for Text-to-Image Evaluation Jiang, Kaiyuan Sun, Ruoxi Cao, Ying Xu, Yuqi Zhang, Xinran Guo, Junyan Deng, ChengSheng Computer Vision and Pattern Recognition We present VISTAR, a user-centric, multi-dimensional benchmark for text-to-image (T2I) evaluation that addresses the limitations of existing metrics. VISTAR introduces a two-tier hybrid paradigm: it employs deterministic, scriptable metrics for physically quantifiable attributes (e.g., text rendering, lighting) and a novel Hierarchical Weighted P/N Questioning (HWPQ) scheme that uses constrained vision-language models to assess abstract semantics (e.g., style fusion, cultural fidelity). Grounded in a Delphi study with 120 experts, we defined seven user roles and nine evaluation angles to construct the benchmark, which comprises 2,845 prompts validated by over 15,000 human pairwise comparisons. Our metrics achieve high human alignment (>75%), with the HWPQ scheme reaching 85.9% accuracy on abstract semantics, significantly outperforming VQA baselines. Comprehensive evaluation of state-of-the-art models reveals no universal champion, as role-weighted scores reorder rankings and provide actionable guidance for domain-specific deployment. All resources are publicly released to foster reproducible T2I assessment. |
| title | VISTAR:A User-Centric and Role-Driven Benchmark for Text-to-Image Evaluation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.06152 |