Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.16310 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914928751804416 |
|---|---|
| author | Tang, Luyao Yuan, Yuxuan Chen, Chaoqi Huang, Kunze Ding, Xinghao Huang, Yue |
| author_facet | Tang, Luyao Yuan, Yuxuan Chen, Chaoqi Huang, Kunze Ding, Xinghao Huang, Yue |
| contents | Foundation models have made incredible strides in achieving zero-shot or few-shot generalization, leveraging prompt engineering to mimic the problem-solving approach of human intelligence. However, when it comes to some foundation models like Segment Anything, there is still a challenge in performing well on out-of-distribution data, including camouflaged and medical images. Inconsistent prompting strategies during fine-tuning and testing further compound the issue, leading to decreased performance. Drawing inspiration from how human cognition processes new environments, we introduce SlotSAM, a method that reconstructs features from the encoder in a self-supervised manner to create object-centric representations. These representations are then integrated into the foundation model, bolstering its object-level perceptual capabilities while reducing the impact of distribution-related variables. The beauty of SlotSAM lies in its simplicity and adaptability to various tasks, making it a versatile solution that significantly enhances the generalization abilities of foundation models. Through limited parameter fine-tuning in a bootstrap manner, our approach paves the way for improved generalization in novel environments. The code is available at github.com/lytang63/SlotSAM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16310 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bootstrap Segmentation Foundation Model under Distribution Shift via Object-Centric Learning Tang, Luyao Yuan, Yuxuan Chen, Chaoqi Huang, Kunze Ding, Xinghao Huang, Yue Computer Vision and Pattern Recognition Foundation models have made incredible strides in achieving zero-shot or few-shot generalization, leveraging prompt engineering to mimic the problem-solving approach of human intelligence. However, when it comes to some foundation models like Segment Anything, there is still a challenge in performing well on out-of-distribution data, including camouflaged and medical images. Inconsistent prompting strategies during fine-tuning and testing further compound the issue, leading to decreased performance. Drawing inspiration from how human cognition processes new environments, we introduce SlotSAM, a method that reconstructs features from the encoder in a self-supervised manner to create object-centric representations. These representations are then integrated into the foundation model, bolstering its object-level perceptual capabilities while reducing the impact of distribution-related variables. The beauty of SlotSAM lies in its simplicity and adaptability to various tasks, making it a versatile solution that significantly enhances the generalization abilities of foundation models. Through limited parameter fine-tuning in a bootstrap manner, our approach paves the way for improved generalization in novel environments. The code is available at github.com/lytang63/SlotSAM. |
| title | Bootstrap Segmentation Foundation Model under Distribution Shift via Object-Centric Learning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.16310 |