EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929761118322688 |
|---|---|
| author | Wang, Guankun Bai, Long Wang, Junyi Yuan, Kun Li, Zhen Jiang, Tianxu He, Xiting Wu, Jinlin Chen, Zhen Lei, Zhen Liu, Hongbin Wang, Jiazheng Zhang, Fan Padoy, Nicolas Navab, Nassir Ren, Hongliang |
| author_facet | Wang, Guankun Bai, Long Wang, Junyi Yuan, Kun Li, Zhen Jiang, Tianxu He, Xiting Wu, Jinlin Chen, Zhen Lei, Zhen Liu, Hongbin Wang, Jiazheng Zhang, Fan Padoy, Nicolas Navab, Nassir Ren, Hongliang |
| contents | Recently, Multimodal Large Language Models (MLLMs) have demonstrated their immense potential in computer-aided diagnosis and decision-making. In the context of robotic-assisted surgery, MLLMs can serve as effective tools for surgical training and guidance. However, there is still a lack of MLLMs specialized for surgical scene understanding in clinical applications. In this work, we introduce EndoChat to address various dialogue paradigms and subtasks in surgical scene understanding that surgeons encounter. To train our EndoChat, we construct the Surg-396K dataset through a novel pipeline that systematically extracts surgical information and generates structured annotations based on collected large-scale endoscopic surgery datasets. Furthermore, we introduce a multi-scale visual token interaction mechanism and a visual contrast-based reasoning mechanism to enhance the model's representation learning and reasoning capabilities. Our model achieves state-of-the-art performance across five dialogue paradigms and eight surgical scene understanding tasks. Additionally, we conduct evaluations with professional surgeons, most of whom provide positive feedback on collaborating with EndoChat. Overall, these results demonstrate that our EndoChat has great potential to significantly advance training and automation in robotic-assisted surgery. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_11347 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery Wang, Guankun Bai, Long Wang, Junyi Yuan, Kun Li, Zhen Jiang, Tianxu He, Xiting Wu, Jinlin Chen, Zhen Lei, Zhen Liu, Hongbin Wang, Jiazheng Zhang, Fan Padoy, Nicolas Navab, Nassir Ren, Hongliang Computer Vision and Pattern Recognition Recently, Multimodal Large Language Models (MLLMs) have demonstrated their immense potential in computer-aided diagnosis and decision-making. In the context of robotic-assisted surgery, MLLMs can serve as effective tools for surgical training and guidance. However, there is still a lack of MLLMs specialized for surgical scene understanding in clinical applications. In this work, we introduce EndoChat to address various dialogue paradigms and subtasks in surgical scene understanding that surgeons encounter. To train our EndoChat, we construct the Surg-396K dataset through a novel pipeline that systematically extracts surgical information and generates structured annotations based on collected large-scale endoscopic surgery datasets. Furthermore, we introduce a multi-scale visual token interaction mechanism and a visual contrast-based reasoning mechanism to enhance the model's representation learning and reasoning capabilities. Our model achieves state-of-the-art performance across five dialogue paradigms and eight surgical scene understanding tasks. Additionally, we conduct evaluations with professional surgeons, most of whom provide positive feedback on collaborating with EndoChat. Overall, these results demonstrate that our EndoChat has great potential to significantly advance training and automation in robotic-assisted surgery. |
| title | EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.11347 |