SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908273230217216 |
|---|---|
| author | Koley, Subhadeep Dutta, Tapas Kumar Sain, Aneeshan Chowdhury, Pinaki Nath Bhunia, Ayan Kumar Song, Yi-Zhe |
| author_facet | Koley, Subhadeep Dutta, Tapas Kumar Sain, Aneeshan Chowdhury, Pinaki Nath Bhunia, Ayan Kumar Song, Yi-Zhe |
| contents | While foundation models have revolutionised computer vision, their effectiveness for sketch understanding remains limited by the unique challenges of abstract, sparse visual inputs. Through systematic analysis, we uncover two fundamental limitations: Stable Diffusion (SD) struggles to extract meaningful features from abstract sketches (unlike its success with photos), and exhibits a pronounced frequency-domain bias that suppresses essential low-frequency components needed for sketch understanding. Rather than costly retraining, we address these limitations by strategically combining SD with CLIP, whose strong semantic understanding naturally compensates for SD's spatial-frequency biases. By dynamically injecting CLIP features into SD's denoising process and adaptively aggregating features across semantic levels, our method achieves state-of-the-art performance in sketch retrieval (+3.35%), recognition (+1.06%), segmentation (+29.42%), and correspondence learning (+21.22%), demonstrating the first truly universal sketch feature representation in the era of foundation models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_14129 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models Koley, Subhadeep Dutta, Tapas Kumar Sain, Aneeshan Chowdhury, Pinaki Nath Bhunia, Ayan Kumar Song, Yi-Zhe Computer Vision and Pattern Recognition While foundation models have revolutionised computer vision, their effectiveness for sketch understanding remains limited by the unique challenges of abstract, sparse visual inputs. Through systematic analysis, we uncover two fundamental limitations: Stable Diffusion (SD) struggles to extract meaningful features from abstract sketches (unlike its success with photos), and exhibits a pronounced frequency-domain bias that suppresses essential low-frequency components needed for sketch understanding. Rather than costly retraining, we address these limitations by strategically combining SD with CLIP, whose strong semantic understanding naturally compensates for SD's spatial-frequency biases. By dynamically injecting CLIP features into SD's denoising process and adaptively aggregating features across semantic levels, our method achieves state-of-the-art performance in sketch retrieval (+3.35%), recognition (+1.06%), segmentation (+29.42%), and correspondence learning (+21.22%), demonstrating the first truly universal sketch feature representation in the era of foundation models. |
| title | SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.14129 |