Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrieval
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910292993114112 |
|---|---|
| author | Lyou, Eunyi Lee, Doyeon Kim, Jooeun Lee, Joonseok |
| author_facet | Lyou, Eunyi Lee, Doyeon Kim, Jooeun Lee, Joonseok |
| contents | Zero-shot learning offers an efficient solution for a machine learning model to treat unseen categories, avoiding exhaustive data collection. Zero-shot Sketch-based Image Retrieval (ZS-SBIR) simulates real-world scenarios where it is hard and costly to collect paired sketch-photo samples. We propose a novel framework that indirectly aligns sketches and photos by contrasting them through texts, removing the necessity of access to sketch-photo pairs. With an explicit modality encoding learned from data, our approach disentangles modality-agnostic semantics from modality-specific information, bridging the modality gap and enabling effective cross-modal content retrieval within a joint latent space. From comprehensive experiments, we verify the efficacy of the proposed model on ZS-SBIR, and it can be also applied to generalized and fine-grained settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_04860 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrieval Lyou, Eunyi Lee, Doyeon Kim, Jooeun Lee, Joonseok Computer Vision and Pattern Recognition Zero-shot learning offers an efficient solution for a machine learning model to treat unseen categories, avoiding exhaustive data collection. Zero-shot Sketch-based Image Retrieval (ZS-SBIR) simulates real-world scenarios where it is hard and costly to collect paired sketch-photo samples. We propose a novel framework that indirectly aligns sketches and photos by contrasting them through texts, removing the necessity of access to sketch-photo pairs. With an explicit modality encoding learned from data, our approach disentangles modality-agnostic semantics from modality-specific information, bridging the modality gap and enabling effective cross-modal content retrieval within a joint latent space. From comprehensive experiments, we verify the efficacy of the proposed model on ZS-SBIR, and it can be also applied to generalized and fine-grained settings. |
| title | Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrieval |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.04860 |