All-in-One: Transferring Vision Foundation Models into Stereo Matching
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909426571542528 |
|---|---|
| author | Zhou, Jingyi Zhang, Haoyu Yuan, Jiakang Ye, Peng Chen, Tao Jiang, Hao Chen, Meiya Zhang, Yangyang |
| author_facet | Zhou, Jingyi Zhang, Haoyu Yuan, Jiakang Ye, Peng Chen, Tao Jiang, Hao Chen, Meiya Zhang, Yangyang |
| contents | As a fundamental vision task, stereo matching has made remarkable progress. While recent iterative optimization-based methods have achieved promising performance, their feature extraction capabilities still have room for improvement. Inspired by the ability of vision foundation models (VFMs) to extract general representations, in this work, we propose AIO-Stereo which can flexibly select and transfer knowledge from multiple heterogeneous VFMs to a single stereo matching model. To better reconcile features between heterogeneous VFMs and the stereo matching model and fully exploit prior knowledge from VFMs, we proposed a dual-level feature utilization mechanism that aligns heterogeneous features and transfers multi-level knowledge. Based on the mechanism, a dual-level selective knowledge transfer module is designed to selectively transfer knowledge and integrate the advantages of multiple VFMs. Experimental results show that AIO-Stereo achieves start-of-the-art performance on multiple datasets and ranks $1^{st}$ on the Middlebury dataset and outperforms all the published work on the ETH3D benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_09912 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | All-in-One: Transferring Vision Foundation Models into Stereo Matching Zhou, Jingyi Zhang, Haoyu Yuan, Jiakang Ye, Peng Chen, Tao Jiang, Hao Chen, Meiya Zhang, Yangyang Computer Vision and Pattern Recognition As a fundamental vision task, stereo matching has made remarkable progress. While recent iterative optimization-based methods have achieved promising performance, their feature extraction capabilities still have room for improvement. Inspired by the ability of vision foundation models (VFMs) to extract general representations, in this work, we propose AIO-Stereo which can flexibly select and transfer knowledge from multiple heterogeneous VFMs to a single stereo matching model. To better reconcile features between heterogeneous VFMs and the stereo matching model and fully exploit prior knowledge from VFMs, we proposed a dual-level feature utilization mechanism that aligns heterogeneous features and transfers multi-level knowledge. Based on the mechanism, a dual-level selective knowledge transfer module is designed to selectively transfer knowledge and integrate the advantages of multiple VFMs. Experimental results show that AIO-Stereo achieves start-of-the-art performance on multiple datasets and ranks $1^{st}$ on the Middlebury dataset and outperforms all the published work on the ETH3D benchmark. |
| title | All-in-One: Transferring Vision Foundation Models into Stereo Matching |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.09912 |