CNS-Obsidian: A Neurosurgical Vision-Language Model Built From Scientific Publications
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918216756887552 |
|---|---|
| author | Alyakin, Anton Stryker, Jaden Alber, Daniel Alexander Lee, Jin Vivian Sangwon, Karl L. Duderstadt, Brandon Save, Akshay Kurland, David Frome, Spencer Singh, Shrutika Zhang, Jeff Yang, Eunice Park, Ki Yun Orillac, Cordelia Valliani, Aly A. Neifert, Sean Liu, Albert Patel, Aneek Livia, Christopher Lau, Darryl Laufer, Ilya Rozman, Peter A. Hidalgo, Eveline Teresa Riina, Howard Feng, Rui Hollon, Todd Aphinyanaphongs, Yindalon Golfinos, John G. Snyder, Laura Leuthardt, Eric Kondziolka, Douglas Oermann, Eric Karl |
| author_facet | Alyakin, Anton Stryker, Jaden Alber, Daniel Alexander Lee, Jin Vivian Sangwon, Karl L. Duderstadt, Brandon Save, Akshay Kurland, David Frome, Spencer Singh, Shrutika Zhang, Jeff Yang, Eunice Park, Ki Yun Orillac, Cordelia Valliani, Aly A. Neifert, Sean Liu, Albert Patel, Aneek Livia, Christopher Lau, Darryl Laufer, Ilya Rozman, Peter A. Hidalgo, Eveline Teresa Riina, Howard Feng, Rui Hollon, Todd Aphinyanaphongs, Yindalon Golfinos, John G. Snyder, Laura Leuthardt, Eric Kondziolka, Douglas Oermann, Eric Karl |
| contents | General-purpose VLMs demonstrate impressive capabilities, but their opaque training on uncurated internet data poses critical limitations for high-stakes decision-making, such as in neurosurgery. We present CNS-Obsidian, a neurosurgical VLM trained on peer-reviewed literature, and demonstrate its clinical utility versus GPT-4o in a real-world setting. We compiled 23,984 articles from Neurosurgery Publications journals, yielding 78,853 figures and captions. Using GPT-4o and Claude Sonnet-3.5, we converted these into 263,064 training samples across three formats: instruction fine-tuning, multiple-choice questions, and differential diagnosis. We trained CNS-Obsidian, a fine-tune of the 34-billion parameter LLaVA-Next model. In a blinded, randomized trial at NYU Langone Health (Aug 30-Nov 30, 2024), neurosurgery consultations were assigned to either CNS-Obsidian or a HIPAA-compliant GPT-4o endpoint as diagnostic co-pilot after consultations. Primary outcomes were diagnostic helpfulness and accuracy, assessed via user ratings and presence of correct diagnosis within the VLM-provided differential. CNS-Obsidian matched GPT-4o on synthetic questions (76.13% vs 77.54%, p=0.235), but only achieved 46.81% accuracy on human-generated questions versus GPT-4o's 65.70% (p<10-15). In the randomized trial, 70 consultations were evaluated (32 CNS-Obsidian, 38 GPT-4o) from 959 total consults (7.3% utilization). CNS-Obsidian received positive ratings in 40.62% of cases versus 57.89% for GPT-4o (p=0.230). Both models included correct diagnosis in approximately 60% of cases (59.38% vs 65.79%, p=0.626). Domain-specific VLMs trained on curated scientific literature can approach frontier model performance despite being orders of magnitude smaller and less expensive to train. This establishes a transparent framework for scientific communities to build specialized AI models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19546 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CNS-Obsidian: A Neurosurgical Vision-Language Model Built From Scientific Publications Alyakin, Anton Stryker, Jaden Alber, Daniel Alexander Lee, Jin Vivian Sangwon, Karl L. Duderstadt, Brandon Save, Akshay Kurland, David Frome, Spencer Singh, Shrutika Zhang, Jeff Yang, Eunice Park, Ki Yun Orillac, Cordelia Valliani, Aly A. Neifert, Sean Liu, Albert Patel, Aneek Livia, Christopher Lau, Darryl Laufer, Ilya Rozman, Peter A. Hidalgo, Eveline Teresa Riina, Howard Feng, Rui Hollon, Todd Aphinyanaphongs, Yindalon Golfinos, John G. Snyder, Laura Leuthardt, Eric Kondziolka, Douglas Oermann, Eric Karl Artificial Intelligence Computation and Language Human-Computer Interaction General-purpose VLMs demonstrate impressive capabilities, but their opaque training on uncurated internet data poses critical limitations for high-stakes decision-making, such as in neurosurgery. We present CNS-Obsidian, a neurosurgical VLM trained on peer-reviewed literature, and demonstrate its clinical utility versus GPT-4o in a real-world setting. We compiled 23,984 articles from Neurosurgery Publications journals, yielding 78,853 figures and captions. Using GPT-4o and Claude Sonnet-3.5, we converted these into 263,064 training samples across three formats: instruction fine-tuning, multiple-choice questions, and differential diagnosis. We trained CNS-Obsidian, a fine-tune of the 34-billion parameter LLaVA-Next model. In a blinded, randomized trial at NYU Langone Health (Aug 30-Nov 30, 2024), neurosurgery consultations were assigned to either CNS-Obsidian or a HIPAA-compliant GPT-4o endpoint as diagnostic co-pilot after consultations. Primary outcomes were diagnostic helpfulness and accuracy, assessed via user ratings and presence of correct diagnosis within the VLM-provided differential. CNS-Obsidian matched GPT-4o on synthetic questions (76.13% vs 77.54%, p=0.235), but only achieved 46.81% accuracy on human-generated questions versus GPT-4o's 65.70% (p<10-15). In the randomized trial, 70 consultations were evaluated (32 CNS-Obsidian, 38 GPT-4o) from 959 total consults (7.3% utilization). CNS-Obsidian received positive ratings in 40.62% of cases versus 57.89% for GPT-4o (p=0.230). Both models included correct diagnosis in approximately 60% of cases (59.38% vs 65.79%, p=0.626). Domain-specific VLMs trained on curated scientific literature can approach frontier model performance despite being orders of magnitude smaller and less expensive to train. This establishes a transparent framework for scientific communities to build specialized AI models. |
| title | CNS-Obsidian: A Neurosurgical Vision-Language Model Built From Scientific Publications |
| topic | Artificial Intelligence Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2502.19546 |