AddressCLIP: Empowering Vision-Language Models for City-wide Image Address Localization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929417570222080 |
|---|---|
| author | Xu, Shixiong Zhang, Chenghao Fan, Lubin Meng, Gaofeng Xiang, Shiming Ye, Jieping |
| author_facet | Xu, Shixiong Zhang, Chenghao Fan, Lubin Meng, Gaofeng Xiang, Shiming Ye, Jieping |
| contents | In this study, we introduce a new problem raised by social media and photojournalism, named Image Address Localization (IAL), which aims to predict the readable textual address where an image was taken. Existing two-stage approaches involve predicting geographical coordinates and converting them into human-readable addresses, which can lead to ambiguity and be resource-intensive. In contrast, we propose an end-to-end framework named AddressCLIP to solve the problem with more semantics, consisting of two key ingredients: i) image-text alignment to align images with addresses and scene captions by contrastive learning, and ii) image-geography matching to constrain image features with the spatial distance in terms of manifold learning. Additionally, we have built three datasets from Pittsburgh and San Francisco on different scales specifically for the IAL problem. Experiments demonstrate that our approach achieves compelling performance on the proposed datasets and outperforms representative transfer learning methods for vision-language models. Furthermore, extensive ablations and visualizations exhibit the effectiveness of the proposed method. The datasets and source code are available at https://github.com/xsx1001/AddressCLIP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_08156 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AddressCLIP: Empowering Vision-Language Models for City-wide Image Address Localization Xu, Shixiong Zhang, Chenghao Fan, Lubin Meng, Gaofeng Xiang, Shiming Ye, Jieping Computer Vision and Pattern Recognition In this study, we introduce a new problem raised by social media and photojournalism, named Image Address Localization (IAL), which aims to predict the readable textual address where an image was taken. Existing two-stage approaches involve predicting geographical coordinates and converting them into human-readable addresses, which can lead to ambiguity and be resource-intensive. In contrast, we propose an end-to-end framework named AddressCLIP to solve the problem with more semantics, consisting of two key ingredients: i) image-text alignment to align images with addresses and scene captions by contrastive learning, and ii) image-geography matching to constrain image features with the spatial distance in terms of manifold learning. Additionally, we have built three datasets from Pittsburgh and San Francisco on different scales specifically for the IAL problem. Experiments demonstrate that our approach achieves compelling performance on the proposed datasets and outperforms representative transfer learning methods for vision-language models. Furthermore, extensive ablations and visualizations exhibit the effectiveness of the proposed method. The datasets and source code are available at https://github.com/xsx1001/AddressCLIP. |
| title | AddressCLIP: Empowering Vision-Language Models for City-wide Image Address Localization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.08156 |