LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917115051638784 |
|---|---|
| author | Hirota, Yusuke Li, Boyi Hachiuma, Ryo Wu, Yueh-Hua Ivanovic, Boris Nakashima, Yuta Pavone, Marco Choi, Yejin Wang, Yu-Chiang Frank Yang, Chao-Han Huck |
| author_facet | Hirota, Yusuke Li, Boyi Hachiuma, Ryo Wu, Yueh-Hua Ivanovic, Boris Nakashima, Yuta Pavone, Marco Choi, Yejin Wang, Yu-Chiang Frank Yang, Chao-Han Huck |
| contents | Large Vision-Language Models (LVLMs) have transformed image captioning, shifting from concise captions to detailed descriptions. We introduce LOTUS, a leaderboard for evaluating detailed captions, addressing three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. LOTUS comprehensively evaluates various aspects, including caption quality (e.g., alignment, descriptiveness), risks (\eg, hallucination), and societal biases (e.g., gender bias) while enabling preference-oriented evaluations by tailoring criteria to diverse user preferences. Our analysis of recent LVLMs reveals no single model excels across all criteria, while correlations emerge between caption detail and bias risks. Preference-oriented evaluations demonstrate that optimal model selection depends on user priorities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_19362 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences Hirota, Yusuke Li, Boyi Hachiuma, Ryo Wu, Yueh-Hua Ivanovic, Boris Nakashima, Yuta Pavone, Marco Choi, Yejin Wang, Yu-Chiang Frank Yang, Chao-Han Huck Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Computers and Society Machine Learning Large Vision-Language Models (LVLMs) have transformed image captioning, shifting from concise captions to detailed descriptions. We introduce LOTUS, a leaderboard for evaluating detailed captions, addressing three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. LOTUS comprehensively evaluates various aspects, including caption quality (e.g., alignment, descriptiveness), risks (\eg, hallucination), and societal biases (e.g., gender bias) while enabling preference-oriented evaluations by tailoring criteria to diverse user preferences. Our analysis of recent LVLMs reveals no single model excels across all criteria, while correlations emerge between caption detail and bias risks. Preference-oriented evaluations demonstrate that optimal model selection depends on user priorities. |
| title | LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Computers and Society Machine Learning |
| url | https://arxiv.org/abs/2507.19362 |