Zero-Shot Pupil Segmentation with SAM 2: A Case Study of Over 14 Million Images
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929673315811328 |
|---|---|
| author | Maquiling, Virmarie Byrne, Sean Anthony Niehorster, Diederick C. Carminati, Marco Kasneci, Enkelejda |
| author_facet | Maquiling, Virmarie Byrne, Sean Anthony Niehorster, Diederick C. Carminati, Marco Kasneci, Enkelejda |
| contents | We explore the transformative potential of SAM 2, a vision foundation model, in advancing gaze estimation and eye tracking technologies. By significantly reducing annotation time, lowering technical barriers through its ease of deployment, and enhancing segmentation accuracy, SAM 2 addresses critical challenges faced by researchers and practitioners. Utilizing its zero-shot segmentation capabilities with minimal user input-a single click per video-we tested SAM 2 on over 14 million eye images from diverse datasets, including virtual reality setups and the world's largest unified dataset recorded using wearable eye trackers. Remarkably, in pupil segmentation tasks, SAM 2 matches the performance of domain-specific models trained solely on eye images, achieving competitive mean Intersection over Union (mIoU) scores of up to 93% without fine-tuning. Additionally, we provide our code and segmentation masks for these widely used datasets to promote further research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08926 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Zero-Shot Pupil Segmentation with SAM 2: A Case Study of Over 14 Million Images Maquiling, Virmarie Byrne, Sean Anthony Niehorster, Diederick C. Carminati, Marco Kasneci, Enkelejda Computer Vision and Pattern Recognition Artificial Intelligence Human-Computer Interaction We explore the transformative potential of SAM 2, a vision foundation model, in advancing gaze estimation and eye tracking technologies. By significantly reducing annotation time, lowering technical barriers through its ease of deployment, and enhancing segmentation accuracy, SAM 2 addresses critical challenges faced by researchers and practitioners. Utilizing its zero-shot segmentation capabilities with minimal user input-a single click per video-we tested SAM 2 on over 14 million eye images from diverse datasets, including virtual reality setups and the world's largest unified dataset recorded using wearable eye trackers. Remarkably, in pupil segmentation tasks, SAM 2 matches the performance of domain-specific models trained solely on eye images, achieving competitive mean Intersection over Union (mIoU) scores of up to 93% without fine-tuning. Additionally, we provide our code and segmentation masks for these widely used datasets to promote further research. |
| title | Zero-Shot Pupil Segmentation with SAM 2: A Case Study of Over 14 Million Images |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2410.08926 |