Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915169159872512 |
|---|---|
| author | Yan, Kun Bouraoui, Zied Wei, Fangyun Xu, Chang Wang, Ping Jameel, Shoaib Schockaert, Steven |
| author_facet | Yan, Kun Bouraoui, Zied Wei, Fangyun Xu, Chang Wang, Ping Jameel, Shoaib Schockaert, Steven |
| contents | The aim of multi-label few-shot image classification (ML-FSIC) is to assign semantic labels to images, in settings where only a small number of training examples are available for each label. A key feature of the multi-label setting is that an image often has several labels, which typically refer to objects appearing in different regions of the image. When estimating label prototypes, in a metric-based setting, it is thus important to determine which regions are relevant for which labels, but the limited amount of training data and the noisy nature of local features make this highly challenging. As a solution, we propose a strategy in which label prototypes are gradually refined. First, we initialize the prototypes using word embeddings, which allows us to leverage prior knowledge about the meaning of the labels. Second, taking advantage of these initial prototypes, we then use a Loss Change Measurement (LCM) strategy to select the local features from the training images (i.e. the support set) that are most likely to be representative of a given label. Third, we construct the final prototype of the label by aggregating these representative local features using a multi-modal cross-interaction mechanism, which again relies on the initial word embedding-based prototypes. Experiments on COCO, PASCAL VOC, NUS-WIDE, and iMaterialist show that our model substantially improves the current state-of-the-art. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13732 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selection Yan, Kun Bouraoui, Zied Wei, Fangyun Xu, Chang Wang, Ping Jameel, Shoaib Schockaert, Steven Computer Vision and Pattern Recognition The aim of multi-label few-shot image classification (ML-FSIC) is to assign semantic labels to images, in settings where only a small number of training examples are available for each label. A key feature of the multi-label setting is that an image often has several labels, which typically refer to objects appearing in different regions of the image. When estimating label prototypes, in a metric-based setting, it is thus important to determine which regions are relevant for which labels, but the limited amount of training data and the noisy nature of local features make this highly challenging. As a solution, we propose a strategy in which label prototypes are gradually refined. First, we initialize the prototypes using word embeddings, which allows us to leverage prior knowledge about the meaning of the labels. Second, taking advantage of these initial prototypes, we then use a Loss Change Measurement (LCM) strategy to select the local features from the training images (i.e. the support set) that are most likely to be representative of a given label. Third, we construct the final prototype of the label by aggregating these representative local features using a multi-modal cross-interaction mechanism, which again relies on the initial word embedding-based prototypes. Experiments on COCO, PASCAL VOC, NUS-WIDE, and iMaterialist show that our model substantially improves the current state-of-the-art. |
| title | Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selection |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.13732 |