Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2602.09252 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917262980546560 |
|---|---|
| author | Lou, Ange Li, Yamin Chang, Qi Xi, Nan Xie, Luyuan Li, Zichao Luan, Tianyu |
| author_facet | Lou, Ange Li, Yamin Chang, Qi Xi, Nan Xie, Luyuan Li, Zichao Luan, Tianyu |
| contents | Surgical image segmentation is essential for robot-assisted surgery and intraoperative guidance. However, existing methods are constrained to predefined categories, produce one-shot predictions without adaptive refinement, and lack mechanisms for clinician interaction. We propose IR-SIS, an iterative refinement system for surgical image segmentation that accepts natural language descriptions. IR-SIS leverages a fine-tuned SAM3 for initial segmentation, employs a Vision-Language Model to detect instruments and assess segmentation quality, and applies an agentic workflow that adaptively selects refinement strategies. The system supports clinician-in-the-loop interaction through natural language feedback. We also construct a multi-granularity language-annotated dataset from EndoVis2017 and EndoVis2018 benchmarks. Experiments demonstrate state-of-the-art performance on both in-domain and out-of-distribution data, with clinician interaction providing additional improvements. Our work establishes the first language-based surgical segmentation framework with adaptive self-refinement capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_09252 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | VLM-Guided Iterative Refinement for Surgical Image Segmentation with Foundation Models Lou, Ange Li, Yamin Chang, Qi Xi, Nan Xie, Luyuan Li, Zichao Luan, Tianyu Computer Vision and Pattern Recognition Artificial Intelligence Multiagent Systems Surgical image segmentation is essential for robot-assisted surgery and intraoperative guidance. However, existing methods are constrained to predefined categories, produce one-shot predictions without adaptive refinement, and lack mechanisms for clinician interaction. We propose IR-SIS, an iterative refinement system for surgical image segmentation that accepts natural language descriptions. IR-SIS leverages a fine-tuned SAM3 for initial segmentation, employs a Vision-Language Model to detect instruments and assess segmentation quality, and applies an agentic workflow that adaptively selects refinement strategies. The system supports clinician-in-the-loop interaction through natural language feedback. We also construct a multi-granularity language-annotated dataset from EndoVis2017 and EndoVis2018 benchmarks. Experiments demonstrate state-of-the-art performance on both in-domain and out-of-distribution data, with clinician interaction providing additional improvements. Our work establishes the first language-based surgical segmentation framework with adaptive self-refinement capabilities. |
| title | VLM-Guided Iterative Refinement for Surgical Image Segmentation with Foundation Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Multiagent Systems |
| url | https://arxiv.org/abs/2602.09252 |