Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.20830 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916923656110080 |
|---|---|
| author | Duangprom, Krit Lambrou, Tryphon Bhattarai, Binod |
| author_facet | Duangprom, Krit Lambrou, Tryphon Bhattarai, Binod |
| contents | This paper presents a novel pipeline for 2D keypoint estima- tion of surgical tools by leveraging Vision Language Models (VLMs) fine- tuned using a low rank adjusting (LoRA) technique. Unlike traditional Convolutional Neural Network (CNN) or Transformer-based approaches, which often suffer from overfitting in small-scale medical datasets, our method harnesses the generalization capabilities of pre-trained VLMs. We carefully design prompts to create an instruction-tuning dataset and use them to align visual features with semantic keypoint descriptions. Experimental results show that with only two epochs of fine tuning, the adapted VLM outperforms the baseline models, demonstrating the ef- fectiveness of LoRA in low-resource scenarios. This approach not only improves keypoint detection performance, but also paves the way for future work in 3D surgical hands and tools pose estimation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_20830 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptation Duangprom, Krit Lambrou, Tryphon Bhattarai, Binod Computer Vision and Pattern Recognition This paper presents a novel pipeline for 2D keypoint estima- tion of surgical tools by leveraging Vision Language Models (VLMs) fine- tuned using a low rank adjusting (LoRA) technique. Unlike traditional Convolutional Neural Network (CNN) or Transformer-based approaches, which often suffer from overfitting in small-scale medical datasets, our method harnesses the generalization capabilities of pre-trained VLMs. We carefully design prompts to create an instruction-tuning dataset and use them to align visual features with semantic keypoint descriptions. Experimental results show that with only two epochs of fine tuning, the adapted VLM outperforms the baseline models, demonstrating the ef- fectiveness of LoRA in low-resource scenarios. This approach not only improves keypoint detection performance, but also paves the way for future work in 3D surgical hands and tools pose estimation. |
| title | Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.20830 |