Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917338329120768 |
|---|---|
| author | Xu, Mengya Shen, Daiyun Zhang, Jie Yip, Hon Chi Gao, Yujia Chen, Cheng Imans, Dillan Long, Yonghao Ye, Yiru Liu, Yixiao Mai, Rongyun Chen, Kai Ren, Hongliang Ban, Yutong Wang, Guangsuo Wong, Francis Ng, Chi-Fai Ngiam, Kee Yuan Taylor, Russell H. Xu, Daguang Jin, Yueming Dou, Qi |
| author_facet | Xu, Mengya Shen, Daiyun Zhang, Jie Yip, Hon Chi Gao, Yujia Chen, Cheng Imans, Dillan Long, Yonghao Ye, Yiru Liu, Yixiao Mai, Rongyun Chen, Kai Ren, Hongliang Ban, Yutong Wang, Guangsuo Wong, Francis Ng, Chi-Fai Ngiam, Kee Yuan Taylor, Russell H. Xu, Daguang Jin, Yueming Dou, Qi |
| contents | Artificial intelligence, imaging, and large language models have the potential to transform surgical practice, training, and automation. Understanding and modeling of basic surgical actions (BSA), the fundamental unit of operation in any surgery, is important to drive the evolution of this field. In this paper, we present a BSA dataset comprising 10 basic actions across 6 surgical specialties with over 11,000 video clips, which is the largest to date. Based on the BSA dataset, we developed a new foundation model that conducts general-purpose recognition of basic actions. Our approach demonstrates robust cross-specialist performance in experiments validated on datasets from different procedural types and various body parts. Furthermore, we demonstrate downstream applications enabled by the BAS foundation model through surgical skill assessment in prostatectomy using domain-specific knowledge, and action planning in cholecystectomy and nephrectomy using large vision-language models. Multinational surgeons' evaluation of the language model's output of the action planning explainable texts demonstrated clinical relevance. These findings indicate that basic surgical actions can be robustly recognized across scenarios, and an accurate BSA understanding model can essentially facilitate complex applications and speed up the realization of surgical superintelligence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_12787 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning Xu, Mengya Shen, Daiyun Zhang, Jie Yip, Hon Chi Gao, Yujia Chen, Cheng Imans, Dillan Long, Yonghao Ye, Yiru Liu, Yixiao Mai, Rongyun Chen, Kai Ren, Hongliang Ban, Yutong Wang, Guangsuo Wong, Francis Ng, Chi-Fai Ngiam, Kee Yuan Taylor, Russell H. Xu, Daguang Jin, Yueming Dou, Qi Computer Vision and Pattern Recognition Artificial intelligence, imaging, and large language models have the potential to transform surgical practice, training, and automation. Understanding and modeling of basic surgical actions (BSA), the fundamental unit of operation in any surgery, is important to drive the evolution of this field. In this paper, we present a BSA dataset comprising 10 basic actions across 6 surgical specialties with over 11,000 video clips, which is the largest to date. Based on the BSA dataset, we developed a new foundation model that conducts general-purpose recognition of basic actions. Our approach demonstrates robust cross-specialist performance in experiments validated on datasets from different procedural types and various body parts. Furthermore, we demonstrate downstream applications enabled by the BAS foundation model through surgical skill assessment in prostatectomy using domain-specific knowledge, and action planning in cholecystectomy and nephrectomy using large vision-language models. Multinational surgeons' evaluation of the language model's output of the action planning explainable texts demonstrated clinical relevance. These findings indicate that basic surgical actions can be robustly recognized across scenarios, and an accurate BSA understanding model can essentially facilitate complex applications and speed up the realization of surgical superintelligence. |
| title | Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.12787 |