Fine-Grained Scene Image Classification with Modality-Agnostic Adapter
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929408113115136 |
|---|---|
| author | Wang, Yiqun Zhou, Zhao Du, Xiangcheng Wu, Xingjiao Zheng, Yingbin Jin, Cheng |
| author_facet | Wang, Yiqun Zhou, Zhao Du, Xiangcheng Wu, Xingjiao Zheng, Yingbin Jin, Cheng |
| contents | When dealing with the task of fine-grained scene image classification, most previous works lay much emphasis on global visual features when doing multi-modal feature fusion. In other words, models are deliberately designed based on prior intuitions about the importance of different modalities. In this paper, we present a new multi-modal feature fusion approach named MAA (Modality-Agnostic Adapter), trying to make the model learn the importance of different modalities in different cases adaptively, without giving a prior setting in the model architecture. More specifically, we eliminate the modal differences in distribution and then use a modality-agnostic Transformer encoder for a semantic-level feature fusion. Our experiments demonstrate that MAA achieves state-of-the-art results on benchmarks by applying the same modalities with previous methods. Besides, it is worth mentioning that new modalities can be easily added when using MAA and further boost the performance. Code is available at https://github.com/quniLcs/MAA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_02769 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fine-Grained Scene Image Classification with Modality-Agnostic Adapter Wang, Yiqun Zhou, Zhao Du, Xiangcheng Wu, Xingjiao Zheng, Yingbin Jin, Cheng Computer Vision and Pattern Recognition When dealing with the task of fine-grained scene image classification, most previous works lay much emphasis on global visual features when doing multi-modal feature fusion. In other words, models are deliberately designed based on prior intuitions about the importance of different modalities. In this paper, we present a new multi-modal feature fusion approach named MAA (Modality-Agnostic Adapter), trying to make the model learn the importance of different modalities in different cases adaptively, without giving a prior setting in the model architecture. More specifically, we eliminate the modal differences in distribution and then use a modality-agnostic Transformer encoder for a semantic-level feature fusion. Our experiments demonstrate that MAA achieves state-of-the-art results on benchmarks by applying the same modalities with previous methods. Besides, it is worth mentioning that new modalities can be easily added when using MAA and further boost the performance. Code is available at https://github.com/quniLcs/MAA. |
| title | Fine-Grained Scene Image Classification with Modality-Agnostic Adapter |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.02769 |