SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916522723639296 |
|---|---|
| author | Chen, Yi-Chia Li, Wei-Hua Sun, Cheng Wang, Yu-Chiang Frank Chen, Chu-Song |
| author_facet | Chen, Yi-Chia Li, Wei-Hua Sun, Cheng Wang, Yu-Chiang Frank Chen, Chu-Song |
| contents | We introduce SAM4MLLM, an innovative approach which integrates the Segment Anything Model (SAM) with Multi-Modal Large Language Models (MLLMs) for pixel-aware tasks. Our method enables MLLMs to learn pixel-level location information without requiring excessive modifications to the existing model architecture or adding specialized tokens. We introduce an inquiry-based approach that can effectively find prompt points for SAM to perform segmentation based on MLLM. It combines detailed visual information with the powerful expressive capabilities of large language models in a unified language-based manner without additional computational overhead in learning. Experimental results on pubic benchmarks demonstrate the effectiveness of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10542 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation Chen, Yi-Chia Li, Wei-Hua Sun, Cheng Wang, Yu-Chiang Frank Chen, Chu-Song Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition We introduce SAM4MLLM, an innovative approach which integrates the Segment Anything Model (SAM) with Multi-Modal Large Language Models (MLLMs) for pixel-aware tasks. Our method enables MLLMs to learn pixel-level location information without requiring excessive modifications to the existing model architecture or adding specialized tokens. We introduce an inquiry-based approach that can effectively find prompt points for SAM to perform segmentation based on MLLM. It combines detailed visual information with the powerful expressive capabilities of large language models in a unified language-based manner without additional computational overhead in learning. Experimental results on pubic benchmarks demonstrate the effectiveness of our approach. |
| title | SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.10542 |