RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918037350776832 |
|---|---|
| author | Wang, Yuchi Cai, Yishuo Ren, Shuhuai Yang, Sihan Yao, Linli Liu, Yuanxin Zhang, Yuanxing Wan, Pengfei Sun, Xu |
| author_facet | Wang, Yuchi Cai, Yishuo Ren, Shuhuai Yang, Sihan Yao, Linli Liu, Yuanxin Zhang, Yuanxing Wan, Pengfei Sun, Xu |
| contents | Image recaptioning is widely used to generate training datasets with enhanced quality for various multimodal tasks. Existing recaptioning methods typically rely on powerful multimodal large language models (MLLMs) to enhance textual descriptions, but often suffer from inaccuracies due to hallucinations and incompleteness caused by missing fine-grained details. To address these limitations, we propose RICO, a novel framework that refines captions through visual reconstruction. Specifically, we leverage a text-to-image model to reconstruct a caption into a reference image, and prompt an MLLM to identify discrepancies between the original and reconstructed images to refine the caption. This process is performed iteratively, further progressively promoting the generation of more faithful and comprehensive descriptions. To mitigate the additional computational cost induced by the iterative process, we introduce RICO-Flash, which learns to generate captions like RICO using DPO. Extensive experiments demonstrate that our approach significantly improves caption accuracy and completeness, outperforms most baselines by approximately 10% on both CapsBench and CompreCap. Code released at https://github.com/wangyuchi369/RICO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22613 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction Wang, Yuchi Cai, Yishuo Ren, Shuhuai Yang, Sihan Yao, Linli Liu, Yuanxin Zhang, Yuanxing Wan, Pengfei Sun, Xu Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Image recaptioning is widely used to generate training datasets with enhanced quality for various multimodal tasks. Existing recaptioning methods typically rely on powerful multimodal large language models (MLLMs) to enhance textual descriptions, but often suffer from inaccuracies due to hallucinations and incompleteness caused by missing fine-grained details. To address these limitations, we propose RICO, a novel framework that refines captions through visual reconstruction. Specifically, we leverage a text-to-image model to reconstruct a caption into a reference image, and prompt an MLLM to identify discrepancies between the original and reconstructed images to refine the caption. This process is performed iteratively, further progressively promoting the generation of more faithful and comprehensive descriptions. To mitigate the additional computational cost induced by the iterative process, we introduce RICO-Flash, which learns to generate captions like RICO using DPO. Extensive experiments demonstrate that our approach significantly improves caption accuracy and completeness, outperforms most baselines by approximately 10% on both CapsBench and CompreCap. Code released at https://github.com/wangyuchi369/RICO. |
| title | RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2505.22613 |