Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910071787618304 |
|---|---|
| author | Robbins, Kevin Liu, Xiaotong Wu, Yu Sun, Le McPeak, Grady Stylianou, Abby Pless, Robert |
| author_facet | Robbins, Kevin Liu, Xiaotong Wu, Yu Sun, Le McPeak, Grady Stylianou, Abby Pless, Robert |
| contents | Vision-Language Models like CLIP create aligned embedding spaces for text and images, making it possible for anyone to build a visual classifier by simply naming the classes they want to distinguish. However, a model that works well in one domain may fail in another, and non-expert users have no straightforward way to assess whether their chosen VLM will work on their problem. We build on prior work using text-only comparisons to evaluate how well a model works for a given natural language task, and explore approaches that also generate synthetic images relevant to that task to evaluate and refine the prediction of zero-shot accuracy. We show that generated imagery to the baseline text-only scores substantially improves the quality of these predictions. Additionally, it gives a user feedback on the kinds of images that were used to make the assessment. Experiments on standard CLIP benchmark datasets demonstrate that the image-based approach helps users predict, without any labeled examples, whether a VLM will be effective for their application. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_17535 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries Robbins, Kevin Liu, Xiaotong Wu, Yu Sun, Le McPeak, Grady Stylianou, Abby Pless, Robert Computer Vision and Pattern Recognition Vision-Language Models like CLIP create aligned embedding spaces for text and images, making it possible for anyone to build a visual classifier by simply naming the classes they want to distinguish. However, a model that works well in one domain may fail in another, and non-expert users have no straightforward way to assess whether their chosen VLM will work on their problem. We build on prior work using text-only comparisons to evaluate how well a model works for a given natural language task, and explore approaches that also generate synthetic images relevant to that task to evaluate and refine the prediction of zero-shot accuracy. We show that generated imagery to the baseline text-only scores substantially improves the quality of these predictions. Additionally, it gives a user feedback on the kinds of images that were used to make the assessment. Experiments on standard CLIP benchmark datasets demonstrate that the image-based approach helps users predict, without any labeled examples, whether a VLM will be effective for their application. |
| title | Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.17535 |