Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910152147337216 |
|---|---|
| author | Prakash, Nirmalendu Oozeer, Narmeen Fatimah Su, Xin Howard, Phillip Shah, Shaan He, Zoe Wanying Wu, Shuang Raval, Shivam Lee, Roy Ka-Wei Khosla, Meenakshi Abdullah, Amir |
| author_facet | Prakash, Nirmalendu Oozeer, Narmeen Fatimah Su, Xin Howard, Phillip Shah, Shaan He, Zoe Wanying Wu, Shuang Raval, Shivam Lee, Roy Ka-Wei Khosla, Meenakshi Abdullah, Amir |
| contents | CLIP retrieval is typically framed as a pointwise similarity problem in a shared embedding space. While CLIP achieves strong global cross-modal alignment, many retrieval failures arise from local geometric inconsistencies: nearby items are incorrectly ordered, leading to systematic confusions (e.g., pentagon vs. hexagon) and produces diffuse, weakly controlled result sets. Prior work largely optimizes for point wise relevance or finetuning to mitigate these problems. We instead view retrieval as a problem of neighborhood alignment. Our work introduces (1) neighborhood-level re-ranking via Hungarian matching, which rewards structural consistency; (2) query-conditioned local steering, where directions derived from contrastive neighborhoods around the query reshape retrieval. We show that these techniques improve retrieval performance on attribute-binding and compositional retrieval tasks. Together, these methods operate on local neighborhoods but serve different roles: re-ranking rewards alignment whereas local steering controls neighborhood structure. This shows that retrieval quality and controllability depend critically on local structure, which can be exploited at inference time without retraining. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_16487 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering Prakash, Nirmalendu Oozeer, Narmeen Fatimah Su, Xin Howard, Phillip Shah, Shaan He, Zoe Wanying Wu, Shuang Raval, Shivam Lee, Roy Ka-Wei Khosla, Meenakshi Abdullah, Amir Computer Vision and Pattern Recognition Artificial Intelligence CLIP retrieval is typically framed as a pointwise similarity problem in a shared embedding space. While CLIP achieves strong global cross-modal alignment, many retrieval failures arise from local geometric inconsistencies: nearby items are incorrectly ordered, leading to systematic confusions (e.g., pentagon vs. hexagon) and produces diffuse, weakly controlled result sets. Prior work largely optimizes for point wise relevance or finetuning to mitigate these problems. We instead view retrieval as a problem of neighborhood alignment. Our work introduces (1) neighborhood-level re-ranking via Hungarian matching, which rewards structural consistency; (2) query-conditioned local steering, where directions derived from contrastive neighborhoods around the query reshape retrieval. We show that these techniques improve retrieval performance on attribute-binding and compositional retrieval tasks. Together, these methods operate on local neighborhoods but serve different roles: re-ranking rewards alignment whereas local steering controls neighborhood structure. This shows that retrieval quality and controllability depend critically on local structure, which can be exploited at inference time without retraining. |
| title | Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2604.16487 |