ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910084101046272 |
|---|---|
| author | Sani, Samin Mahdizadeh Ku, Max Jamali, Nima Sani, Matina Mahdizadeh Khoshtab, Paria Sun, Wei-Chieh Fazel, Parnian Tam, Zhi Rui Chong, Thomas Chan, Edisy Kin Wai Tsang, Donald Wai Tong Hsu, Chiao-Wei Lam, Ting Wai Ng, Ho Yin Sam Chu, Chiafeng Mak, Chak-Wing Wu, Keming Wong, Hiu Tung Ho, Yik Chun Ruan, Chi Li, Zhuofeng Fang, I-Sheng Yeh, Shih-Ying Cheng, Ho Kei Nie, Ping Chen, Wenhu |
| author_facet | Sani, Samin Mahdizadeh Ku, Max Jamali, Nima Sani, Matina Mahdizadeh Khoshtab, Paria Sun, Wei-Chieh Fazel, Parnian Tam, Zhi Rui Chong, Thomas Chan, Edisy Kin Wai Tsang, Donald Wai Tong Hsu, Chiao-Wei Lam, Ting Wai Ng, Ho Yin Sam Chu, Chiafeng Mak, Chak-Wing Wu, Keming Wong, Hiu Tung Ho, Yik Chun Ruan, Chi Li, Zhuofeng Fang, I-Sheng Yeh, Shih-Ying Cheng, Ho Kei Nie, Ping Chen, Wenhu |
| contents | Advances in diffusion, autoregressive, and hybrid models have enabled high-quality image synthesis for tasks such as text-to-image, editing, and reference-guided composition. Yet, existing benchmarks remain limited, either focus on isolated tasks, cover only narrow domains, or provide opaque scores without explaining failure modes. We introduce \textbf{ImagenWorld}, a benchmark of 3.6K condition sets spanning six core tasks (generation and editing, with single or multiple references) and six topical domains (artworks, photorealistic images, information graphics, textual graphics, computer graphics, and screenshots). The benchmark is supported by 20K fine-grained human annotations and an explainable evaluation schema that tags localized object-level and segment-level errors, complementing automated VLM-based metrics. Our large-scale evaluation of 14 models yields several insights: (1) models typically struggle more in editing tasks than in generation tasks, especially in local edits. (2) models excel in artistic and photorealistic settings but struggle with symbolic and text-heavy domains such as screenshots and information graphics. (3) closed-source systems lead overall, while targeted data curation (e.g., Qwen-Image) narrows the gap in text-heavy cases. (4) modern VLM-based metrics achieve Kendall accuracies up to 0.79, approximating human ranking, but fall short of fine-grained, explainable error attribution. ImagenWorld provides both a rigorous benchmark and a diagnostic tool to advance robust image generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_27862 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks Sani, Samin Mahdizadeh Ku, Max Jamali, Nima Sani, Matina Mahdizadeh Khoshtab, Paria Sun, Wei-Chieh Fazel, Parnian Tam, Zhi Rui Chong, Thomas Chan, Edisy Kin Wai Tsang, Donald Wai Tong Hsu, Chiao-Wei Lam, Ting Wai Ng, Ho Yin Sam Chu, Chiafeng Mak, Chak-Wing Wu, Keming Wong, Hiu Tung Ho, Yik Chun Ruan, Chi Li, Zhuofeng Fang, I-Sheng Yeh, Shih-Ying Cheng, Ho Kei Nie, Ping Chen, Wenhu Graphics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Advances in diffusion, autoregressive, and hybrid models have enabled high-quality image synthesis for tasks such as text-to-image, editing, and reference-guided composition. Yet, existing benchmarks remain limited, either focus on isolated tasks, cover only narrow domains, or provide opaque scores without explaining failure modes. We introduce \textbf{ImagenWorld}, a benchmark of 3.6K condition sets spanning six core tasks (generation and editing, with single or multiple references) and six topical domains (artworks, photorealistic images, information graphics, textual graphics, computer graphics, and screenshots). The benchmark is supported by 20K fine-grained human annotations and an explainable evaluation schema that tags localized object-level and segment-level errors, complementing automated VLM-based metrics. Our large-scale evaluation of 14 models yields several insights: (1) models typically struggle more in editing tasks than in generation tasks, especially in local edits. (2) models excel in artistic and photorealistic settings but struggle with symbolic and text-heavy domains such as screenshots and information graphics. (3) closed-source systems lead overall, while targeted data curation (e.g., Qwen-Image) narrows the gap in text-heavy cases. (4) modern VLM-based metrics achieve Kendall accuracies up to 0.79, approximating human ranking, but fall short of fine-grained, explainable error attribution. ImagenWorld provides both a rigorous benchmark and a diagnostic tool to advance robust image generation. |
| title | ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks |
| topic | Graphics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2603.27862 |