SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909540494082048 |
|---|---|
| author | Koley, Subhadeep Gajjala, Viswanatha Reddy Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Bhunia, Ayan Kumar Song, Yi-Zhe |
| author_facet | Koley, Subhadeep Gajjala, Viswanatha Reddy Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Bhunia, Ayan Kumar Song, Yi-Zhe |
| contents | We introduce SketchYourSeg, a novel framework that establishes freehand sketches as a powerful query modality for subjective image segmentation across entire galleries through a single exemplar sketch. Unlike text prompts that struggle with spatial specificity or interactive methods confined to single-image operations, sketches naturally combine semantic intent with structural precision. This unique dual encoding enables precise visual disambiguation for segmentation tasks where text descriptions would be cumbersome or ambiguous -- such as distinguishing between visually similar instances, specifying exact part boundaries, or indicating spatial relationships in composed concepts. Our approach addresses three fundamental challenges: (i) eliminating the need for pixel-perfect annotation masks during training with a mask-free framework; (ii) creating a synergistic relationship between sketch-based image retrieval (SBIR) models and foundation models (CLIP/DINOv2) where the former provides training signals while the latter generates masks; and (iii) enabling multi-granular segmentation capabilities through purpose-made sketch augmentation strategies. Our extensive evaluations demonstrate superior performance over existing approaches across diverse benchmarks, establishing a new paradigm for user-guided image segmentation that balances precision with efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16022 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches Koley, Subhadeep Gajjala, Viswanatha Reddy Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Bhunia, Ayan Kumar Song, Yi-Zhe Computer Vision and Pattern Recognition We introduce SketchYourSeg, a novel framework that establishes freehand sketches as a powerful query modality for subjective image segmentation across entire galleries through a single exemplar sketch. Unlike text prompts that struggle with spatial specificity or interactive methods confined to single-image operations, sketches naturally combine semantic intent with structural precision. This unique dual encoding enables precise visual disambiguation for segmentation tasks where text descriptions would be cumbersome or ambiguous -- such as distinguishing between visually similar instances, specifying exact part boundaries, or indicating spatial relationships in composed concepts. Our approach addresses three fundamental challenges: (i) eliminating the need for pixel-perfect annotation masks during training with a mask-free framework; (ii) creating a synergistic relationship between sketch-based image retrieval (SBIR) models and foundation models (CLIP/DINOv2) where the former provides training signals while the latter generates masks; and (iii) enabling multi-granular segmentation capabilities through purpose-made sketch augmentation strategies. Our extensive evaluations demonstrate superior performance over existing approaches across diverse benchmarks, establishing a new paradigm for user-guided image segmentation that balances precision with efficiency. |
| title | SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.16022 |