Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.16060 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908473209389056 |
|---|---|
| author | Ma, Ziqiao Ding, Jing Zhang, Xuejun Luo, Dezhi Ding, Jiahe Xu, Sihan Huang, Yuchen Peng, Run Chai, Joyce |
| author_facet | Ma, Ziqiao Ding, Jing Zhang, Xuejun Luo, Dezhi Ding, Jiahe Xu, Sihan Huang, Yuchen Peng, Run Chai, Joyce |
| contents | Referring Expression Generation (REG) is a core task for evaluating the pragmatic competence of vision-language systems, requiring not only accurate semantic grounding but also adherence to principles of cooperative communication (Grice, 1975). However, current evaluations of vision-language models (VLMs) often overlook the pragmatic dimension, reducing REG to a region-based captioning task and neglecting Gricean maxims. In this work, we revisit REG from a pragmatic perspective, introducing a new dataset (RefOI) of 1.5k images annotated with both written and spoken referring expressions. Through a systematic evaluation of state-of-the-art VLMs, we identify three key failures of pragmatic competence: (1) failure to uniquely identify the referent, (2) inclusion of excessive or irrelevant information, and (3) misalignment with human pragmatic preference, such as the underuse of minimal spatial cues. We also show that standard automatic evaluations fail to capture these pragmatic violations, reinforcing superficial cues rather than genuine referential success. Our findings call for a renewed focus on pragmatically informed models and evaluation frameworks that align with real human communication. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_16060 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation Ma, Ziqiao Ding, Jing Zhang, Xuejun Luo, Dezhi Ding, Jiahe Xu, Sihan Huang, Yuchen Peng, Run Chai, Joyce Computation and Language Referring Expression Generation (REG) is a core task for evaluating the pragmatic competence of vision-language systems, requiring not only accurate semantic grounding but also adherence to principles of cooperative communication (Grice, 1975). However, current evaluations of vision-language models (VLMs) often overlook the pragmatic dimension, reducing REG to a region-based captioning task and neglecting Gricean maxims. In this work, we revisit REG from a pragmatic perspective, introducing a new dataset (RefOI) of 1.5k images annotated with both written and spoken referring expressions. Through a systematic evaluation of state-of-the-art VLMs, we identify three key failures of pragmatic competence: (1) failure to uniquely identify the referent, (2) inclusion of excessive or irrelevant information, and (3) misalignment with human pragmatic preference, such as the underuse of minimal spatial cues. We also show that standard automatic evaluations fail to capture these pragmatic violations, reinforcing superficial cues rather than genuine referential success. Our findings call for a renewed focus on pragmatically informed models and evaluation frameworks that align with real human communication. |
| title | Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.16060 |