CoReS: Orchestrating the Dance of Reasoning and Segmentation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909250105638912 |
|---|---|
| author | Bao, Xiaoyi Sun, Siyang Ma, Shuailei Zheng, Kecheng Guo, Yuxin Zhao, Guosheng Zheng, Yun Wang, Xingang |
| author_facet | Bao, Xiaoyi Sun, Siyang Ma, Shuailei Zheng, Kecheng Guo, Yuxin Zhao, Guosheng Zheng, Yun Wang, Xingang |
| contents | The reasoning segmentation task, which demands a nuanced comprehension of intricate queries to accurately pinpoint object regions, is attracting increasing attention. However, Multi-modal Large Language Models (MLLM) often find it difficult to accurately localize the objects described in complex reasoning contexts. We believe that the act of reasoning segmentation should mirror the cognitive stages of human visual search, where each step is a progressive refinement of thought toward the final object. Thus we introduce the Chains of Reasoning and Segmenting (CoReS) and find this top-down visual hierarchy indeed enhances the visual search process. Specifically, we propose a dual-chain structure that generates multi-modal, chain-like outputs to aid the segmentation process. Furthermore, to steer the MLLM's outputs into this intended hierarchy, we incorporate in-context inputs as guidance. Extensive experiments demonstrate the superior performance of our CoReS, which surpasses the state-of-the-art method by 6.5\% on the ReasonSeg dataset. Project: https://chain-of-reasoning-and-segmentation.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_05673 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CoReS: Orchestrating the Dance of Reasoning and Segmentation Bao, Xiaoyi Sun, Siyang Ma, Shuailei Zheng, Kecheng Guo, Yuxin Zhao, Guosheng Zheng, Yun Wang, Xingang Computer Vision and Pattern Recognition The reasoning segmentation task, which demands a nuanced comprehension of intricate queries to accurately pinpoint object regions, is attracting increasing attention. However, Multi-modal Large Language Models (MLLM) often find it difficult to accurately localize the objects described in complex reasoning contexts. We believe that the act of reasoning segmentation should mirror the cognitive stages of human visual search, where each step is a progressive refinement of thought toward the final object. Thus we introduce the Chains of Reasoning and Segmenting (CoReS) and find this top-down visual hierarchy indeed enhances the visual search process. Specifically, we propose a dual-chain structure that generates multi-modal, chain-like outputs to aid the segmentation process. Furthermore, to steer the MLLM's outputs into this intended hierarchy, we incorporate in-context inputs as guidance. Extensive experiments demonstrate the superior performance of our CoReS, which surpasses the state-of-the-art method by 6.5\% on the ReasonSeg dataset. Project: https://chain-of-reasoning-and-segmentation.github.io/. |
| title | CoReS: Orchestrating the Dance of Reasoning and Segmentation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.05673 |