Learning Visual Conditioning Tokens to Correct Domain Shift for Fully Test-time Adaptation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917724281634816 |
|---|---|
| author | Tang, Yushun Chen, Shuoshuo Kan, Zhehan Zhang, Yi Guo, Qinghai He, Zhihai |
| author_facet | Tang, Yushun Chen, Shuoshuo Kan, Zhehan Zhang, Yi Guo, Qinghai He, Zhihai |
| contents | Fully test-time adaptation aims to adapt the network model based on sequential analysis of input samples during the inference stage to address the cross-domain performance degradation problem of deep neural networks. This work is based on the following interesting finding: in transformer-based image classification, the class token at the first transformer encoder layer can be learned to capture the domain-specific characteristics of target samples during test-time adaptation. This learned token, when combined with input image patch embeddings, is able to gradually remove the domain-specific information from the feature representations of input samples during the transformer encoding process, thereby significantly improving the test-time adaptation performance of the source model across different domains. We refer to this class token as visual conditioning token (VCT). To successfully learn the VCT, we propose a bi-level learning approach to capture the long-term variations of domain-specific characteristics while accommodating local variations of instance-specific characteristics. Experimental results on the benchmark datasets demonstrate that our proposed bi-level visual conditioning token learning method is able to achieve significantly improved test-time adaptation performance by up to 1.9%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19341 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning Visual Conditioning Tokens to Correct Domain Shift for Fully Test-time Adaptation Tang, Yushun Chen, Shuoshuo Kan, Zhehan Zhang, Yi Guo, Qinghai He, Zhihai Computer Vision and Pattern Recognition Fully test-time adaptation aims to adapt the network model based on sequential analysis of input samples during the inference stage to address the cross-domain performance degradation problem of deep neural networks. This work is based on the following interesting finding: in transformer-based image classification, the class token at the first transformer encoder layer can be learned to capture the domain-specific characteristics of target samples during test-time adaptation. This learned token, when combined with input image patch embeddings, is able to gradually remove the domain-specific information from the feature representations of input samples during the transformer encoding process, thereby significantly improving the test-time adaptation performance of the source model across different domains. We refer to this class token as visual conditioning token (VCT). To successfully learn the VCT, we propose a bi-level learning approach to capture the long-term variations of domain-specific characteristics while accommodating local variations of instance-specific characteristics. Experimental results on the benchmark datasets demonstrate that our proposed bi-level visual conditioning token learning method is able to achieve significantly improved test-time adaptation performance by up to 1.9%. |
| title | Learning Visual Conditioning Tokens to Correct Domain Shift for Fully Test-time Adaptation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.19341 |