FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916685652426752 |
|---|---|
| author | Hsieh, Cheng-Yu Vasu, Pavan Kumar Anasosalu Faghri, Fartash Vemulapalli, Raviteja Li, Chun-Liang Krishna, Ranjay Tuzel, Oncel Pouransari, Hadi |
| author_facet | Hsieh, Cheng-Yu Vasu, Pavan Kumar Anasosalu Faghri, Fartash Vemulapalli, Raviteja Li, Chun-Liang Krishna, Ranjay Tuzel, Oncel Pouransari, Hadi |
| contents | Visual understanding is inherently contextual -- what we focus on in an image depends on the task at hand. For instance, given an image of a person holding a bouquet of flowers, we may focus on either the person such as their clothing, or the type of flowers, depending on the context of interest. Yet, most existing image encoding paradigms represent an image as a fixed, generic feature vector, overlooking the potential needs of prioritizing varying visual information for different downstream use cases. In this work, we introduce FocalLens, a conditional visual encoding method that produces different representations for the same image based on the context of interest, expressed flexibly through natural language. We leverage vision instruction tuning data and contrastively finetune a pretrained vision encoder to take natural language instructions as additional inputs for producing conditional image representations. Extensive experiments validate that conditional image representation from FocalLens better pronounce the visual features of interest compared to generic features produced by standard vision encoders like CLIP. In addition, we show FocalLens further leads to performance improvements on a range of downstream tasks including image-image retrieval, image classification, and image-text retrieval, with an average gain of 5 and 10 points on the challenging SugarCrepe and MMVP-VLM benchmarks, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08368 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations Hsieh, Cheng-Yu Vasu, Pavan Kumar Anasosalu Faghri, Fartash Vemulapalli, Raviteja Li, Chun-Liang Krishna, Ranjay Tuzel, Oncel Pouransari, Hadi Computer Vision and Pattern Recognition Computation and Language Machine Learning Visual understanding is inherently contextual -- what we focus on in an image depends on the task at hand. For instance, given an image of a person holding a bouquet of flowers, we may focus on either the person such as their clothing, or the type of flowers, depending on the context of interest. Yet, most existing image encoding paradigms represent an image as a fixed, generic feature vector, overlooking the potential needs of prioritizing varying visual information for different downstream use cases. In this work, we introduce FocalLens, a conditional visual encoding method that produces different representations for the same image based on the context of interest, expressed flexibly through natural language. We leverage vision instruction tuning data and contrastively finetune a pretrained vision encoder to take natural language instructions as additional inputs for producing conditional image representations. Extensive experiments validate that conditional image representation from FocalLens better pronounce the visual features of interest compared to generic features produced by standard vision encoders like CLIP. In addition, we show FocalLens further leads to performance improvements on a range of downstream tasks including image-image retrieval, image classification, and image-text retrieval, with an average gain of 5 and 10 points on the challenging SugarCrepe and MMVP-VLM benchmarks, respectively. |
| title | FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2504.08368 |