Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.26865 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917358753284096 |
|---|---|
| author | Lin, Fenfen Liu, Yesheng Xu, Haiyu Yue, Chen He, Zheqi Zhao, Mingxuan Chen, Miguel Hu Liu, Jiakang Yao, JG Yang, Xi |
| author_facet | Lin, Fenfen Liu, Yesheng Xu, Haiyu Yue, Chen He, Zheqi Zhao, Mingxuan Chen, Miguel Hu Liu, Jiakang Yao, JG Yang, Xi |
| contents | Reading measurement instruments is effortless for humans and requires relatively little domain expertise, yet it remains surprisingly challenging for current vision-language models (VLMs) as we find in preliminary evaluation. In this work, we introduce MeasureBench, a benchmark on visual measurement reading covering both real-world and synthesized images of various types of measurements, along with an extensible pipeline for data synthesis. Our pipeline procedurally generates a specified type of gauge with controllable visual appearance, enabling scalable variation in key details such as pointers, scales, fonts, lighting, and clutter. Evaluation on popular proprietary and open-weight VLMs shows that even the strongest frontier VLMs struggle with measurement reading in general. We have also conducted preliminary experiments with reinforcement finetuning (RFT) over synthetic data, and find a significant improvement on both in-domain synthetic subset and real-world images. Our analysis highlights a fundamental limitation of current VLMs in fine-grained spatial grounding. We hope this resource and our code releases can help future advances on visually grounded numeracy and precise spatial perception of VLMs, bridging the gap between recognizing numbers and measuring the world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_26865 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench Lin, Fenfen Liu, Yesheng Xu, Haiyu Yue, Chen He, Zheqi Zhao, Mingxuan Chen, Miguel Hu Liu, Jiakang Yao, JG Yang, Xi Computer Vision and Pattern Recognition Artificial Intelligence Reading measurement instruments is effortless for humans and requires relatively little domain expertise, yet it remains surprisingly challenging for current vision-language models (VLMs) as we find in preliminary evaluation. In this work, we introduce MeasureBench, a benchmark on visual measurement reading covering both real-world and synthesized images of various types of measurements, along with an extensible pipeline for data synthesis. Our pipeline procedurally generates a specified type of gauge with controllable visual appearance, enabling scalable variation in key details such as pointers, scales, fonts, lighting, and clutter. Evaluation on popular proprietary and open-weight VLMs shows that even the strongest frontier VLMs struggle with measurement reading in general. We have also conducted preliminary experiments with reinforcement finetuning (RFT) over synthetic data, and find a significant improvement on both in-domain synthetic subset and real-world images. Our analysis highlights a fundamental limitation of current VLMs in fine-grained spatial grounding. We hope this resource and our code releases can help future advances on visually grounded numeracy and precise spatial perception of VLMs, bridging the gap between recognizing numbers and measuring the world. |
| title | Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2510.26865 |