SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909153834827776 |
|---|---|
| author | Hou, Xiaojun Xing, Jiazheng Qian, Yijie Guo, Yaowei Xin, Shuo Chen, Junhao Tang, Kai Wang, Mengmeng Jiang, Zhengkai Liu, Liang Liu, Yong |
| author_facet | Hou, Xiaojun Xing, Jiazheng Qian, Yijie Guo, Yaowei Xin, Shuo Chen, Junhao Tang, Kai Wang, Mengmeng Jiang, Zhengkai Liu, Liang Liu, Yong |
| contents | Multimodal Visual Object Tracking (VOT) has recently gained significant attention due to its robustness. Early research focused on fully fine-tuning RGB-based trackers, which was inefficient and lacked generalized representation due to the scarcity of multimodal data. Therefore, recent studies have utilized prompt tuning to transfer pre-trained RGB-based trackers to multimodal data. However, the modality gap limits pre-trained knowledge recall, and the dominance of the RGB modality persists, preventing the full utilization of information from other modalities. To address these issues, we propose a novel symmetric multimodal tracking framework called SDSTrack. We introduce lightweight adaptation for efficient fine-tuning, which directly transfers the feature extraction ability from RGB to other domains with a small number of trainable parameters and integrates multimodal features in a balanced, symmetric manner. Furthermore, we design a complementary masked patch distillation strategy to enhance the robustness of trackers in complex environments, such as extreme weather, poor imaging, and sensor failure. Extensive experiments demonstrate that SDSTrack outperforms state-of-the-art methods in various multimodal tracking scenarios, including RGB+Depth, RGB+Thermal, and RGB+Event tracking, and exhibits impressive results in extreme conditions. Our source code is available at https://github.com/hoqolo/SDSTrack. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_16002 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking Hou, Xiaojun Xing, Jiazheng Qian, Yijie Guo, Yaowei Xin, Shuo Chen, Junhao Tang, Kai Wang, Mengmeng Jiang, Zhengkai Liu, Liang Liu, Yong Computer Vision and Pattern Recognition Multimodal Visual Object Tracking (VOT) has recently gained significant attention due to its robustness. Early research focused on fully fine-tuning RGB-based trackers, which was inefficient and lacked generalized representation due to the scarcity of multimodal data. Therefore, recent studies have utilized prompt tuning to transfer pre-trained RGB-based trackers to multimodal data. However, the modality gap limits pre-trained knowledge recall, and the dominance of the RGB modality persists, preventing the full utilization of information from other modalities. To address these issues, we propose a novel symmetric multimodal tracking framework called SDSTrack. We introduce lightweight adaptation for efficient fine-tuning, which directly transfers the feature extraction ability from RGB to other domains with a small number of trainable parameters and integrates multimodal features in a balanced, symmetric manner. Furthermore, we design a complementary masked patch distillation strategy to enhance the robustness of trackers in complex environments, such as extreme weather, poor imaging, and sensor failure. Extensive experiments demonstrate that SDSTrack outperforms state-of-the-art methods in various multimodal tracking scenarios, including RGB+Depth, RGB+Thermal, and RGB+Event tracking, and exhibits impressive results in extreme conditions. Our source code is available at https://github.com/hoqolo/SDSTrack. |
| title | SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.16002 |