LLaFS: When Large Language Models Meet Few-Shot Segmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909158747406336 |
|---|---|
| author | Zhu, Lanyun Chen, Tianrun Ji, Deyi Ye, Jieping Liu, Jun |
| author_facet | Zhu, Lanyun Chen, Tianrun Ji, Deyi Ye, Jieping Liu, Jun |
| contents | This paper proposes LLaFS, the first attempt to leverage large language models (LLMs) in few-shot segmentation. In contrast to the conventional few-shot segmentation methods that only rely on the limited and biased information from the annotated support images, LLaFS leverages the vast prior knowledge gained by LLM as an effective supplement and directly uses the LLM to segment images in a few-shot manner. To enable the text-based LLM to handle image-related tasks, we carefully design an input instruction that allows the LLM to produce segmentation results represented as polygons, and propose a region-attribute table to simulate the human visual mechanism and provide multi-modal guidance. We also synthesize pseudo samples and use curriculum learning for pretraining to augment data and achieve better optimization. LLaFS achieves state-of-the-art results on multiple datasets, showing the potential of using LLMs for few-shot computer vision tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_16926 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | LLaFS: When Large Language Models Meet Few-Shot Segmentation Zhu, Lanyun Chen, Tianrun Ji, Deyi Ye, Jieping Liu, Jun Computer Vision and Pattern Recognition This paper proposes LLaFS, the first attempt to leverage large language models (LLMs) in few-shot segmentation. In contrast to the conventional few-shot segmentation methods that only rely on the limited and biased information from the annotated support images, LLaFS leverages the vast prior knowledge gained by LLM as an effective supplement and directly uses the LLM to segment images in a few-shot manner. To enable the text-based LLM to handle image-related tasks, we carefully design an input instruction that allows the LLM to produce segmentation results represented as polygons, and propose a region-attribute table to simulate the human visual mechanism and provide multi-modal guidance. We also synthesize pseudo samples and use curriculum learning for pretraining to augment data and achieve better optimization. LLaFS achieves state-of-the-art results on multiple datasets, showing the potential of using LLMs for few-shot computer vision tasks. |
| title | LLaFS: When Large Language Models Meet Few-Shot Segmentation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.16926 |