Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910362653163520 |
|---|---|
| author | Li, Xinyao Li, Yuke Du, Zhekai Li, Fengling Lu, Ke Li, Jingjing |
| author_facet | Li, Xinyao Li, Yuke Du, Zhekai Li, Fengling Lu, Ke Li, Jingjing |
| contents | Large vision-language models (VLMs) like CLIP have demonstrated good zero-shot learning performance in the unsupervised domain adaptation task. Yet, most transfer approaches for VLMs focus on either the language or visual branches, overlooking the nuanced interplay between both modalities. In this work, we introduce a Unified Modality Separation (UniMoS) framework for unsupervised domain adaptation. Leveraging insights from modality gap studies, we craft a nimble modality separation network that distinctly disentangles CLIP's features into language-associated and vision-associated components. Our proposed Modality-Ensemble Training (MET) method fosters the exchange of modality-agnostic information while maintaining modality-specific nuances. We align features across domains using a modality discriminator. Comprehensive evaluations on three benchmarks reveal our approach sets a new state-of-the-art with minimal computational costs. Code: https://github.com/TL-UESTC/UniMoS |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_06946 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation Li, Xinyao Li, Yuke Du, Zhekai Li, Fengling Lu, Ke Li, Jingjing Computer Vision and Pattern Recognition Large vision-language models (VLMs) like CLIP have demonstrated good zero-shot learning performance in the unsupervised domain adaptation task. Yet, most transfer approaches for VLMs focus on either the language or visual branches, overlooking the nuanced interplay between both modalities. In this work, we introduce a Unified Modality Separation (UniMoS) framework for unsupervised domain adaptation. Leveraging insights from modality gap studies, we craft a nimble modality separation network that distinctly disentangles CLIP's features into language-associated and vision-associated components. Our proposed Modality-Ensemble Training (MET) method fosters the exchange of modality-agnostic information while maintaining modality-specific nuances. We align features across domains using a modality discriminator. Comprehensive evaluations on three benchmarks reveal our approach sets a new state-of-the-art with minimal computational costs. Code: https://github.com/TL-UESTC/UniMoS |
| title | Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.06946 |