NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913848184799232 |
|---|---|
| author | Bercea, Cosmin I. Li, Jun Raffler, Philipp Riedel, Evamaria O. Schmitzer, Lena Kurz, Angela Bitzer, Felix Roßmüller, Paula Canisius, Julian Beyrle, Mirjam L. Liu, Che Bai, Wenjia Kainz, Bernhard Schnabel, Julia A. Wiestler, Benedikt |
| author_facet | Bercea, Cosmin I. Li, Jun Raffler, Philipp Riedel, Evamaria O. Schmitzer, Lena Kurz, Angela Bitzer, Felix Roßmüller, Paula Canisius, Julian Beyrle, Mirjam L. Liu, Che Bai, Wenjia Kainz, Bernhard Schnabel, Julia A. Wiestler, Benedikt |
| contents | In many real-world applications, deployed models encounter inputs that differ from the data seen during training. Out-of-distribution detection identifies whether an input stems from an unseen distribution, while open-world recognition flags such inputs to ensure the system remains robust as ever-emerging, previously $unknown$ categories appear and must be addressed without retraining. Foundation and vision-language models are pre-trained on large and diverse datasets with the expectation of broad generalization across domains, including medical imaging. However, benchmarking these models on test sets with only a few common outlier types silently collapses the evaluation back to a closed-set problem, masking failures on rare or truly novel conditions encountered in clinical use.
We therefore present $NOVA$, a challenging, real-life $evaluation-only$ benchmark of $\sim$900 brain MRI scans that span 281 rare pathologies and heterogeneous acquisition protocols. Each case includes rich clinical narratives and double-blinded expert bounding-box annotations. Together, these enable joint assessment of anomaly localisation, visual captioning, and diagnostic reasoning. Because NOVA is never used for training, it serves as an $extreme$ stress-test of out-of-distribution generalisation: models must bridge a distribution gap both in sample appearance and in semantic space. Baseline results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-72B) reveal substantial performance drops across all tasks, establishing NOVA as a rigorous testbed for advancing models that can detect, localize, and reason about truly unknown anomalies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14064 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI Bercea, Cosmin I. Li, Jun Raffler, Philipp Riedel, Evamaria O. Schmitzer, Lena Kurz, Angela Bitzer, Felix Roßmüller, Paula Canisius, Julian Beyrle, Mirjam L. Liu, Che Bai, Wenjia Kainz, Bernhard Schnabel, Julia A. Wiestler, Benedikt Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning In many real-world applications, deployed models encounter inputs that differ from the data seen during training. Out-of-distribution detection identifies whether an input stems from an unseen distribution, while open-world recognition flags such inputs to ensure the system remains robust as ever-emerging, previously $unknown$ categories appear and must be addressed without retraining. Foundation and vision-language models are pre-trained on large and diverse datasets with the expectation of broad generalization across domains, including medical imaging. However, benchmarking these models on test sets with only a few common outlier types silently collapses the evaluation back to a closed-set problem, masking failures on rare or truly novel conditions encountered in clinical use. We therefore present $NOVA$, a challenging, real-life $evaluation-only$ benchmark of $\sim$900 brain MRI scans that span 281 rare pathologies and heterogeneous acquisition protocols. Each case includes rich clinical narratives and double-blinded expert bounding-box annotations. Together, these enable joint assessment of anomaly localisation, visual captioning, and diagnostic reasoning. Because NOVA is never used for training, it serves as an $extreme$ stress-test of out-of-distribution generalisation: models must bridge a distribution gap both in sample appearance and in semantic space. Baseline results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-72B) reveal substantial performance drops across all tasks, establishing NOVA as a rigorous testbed for advancing models that can detect, localize, and reason about truly unknown anomalies. |
| title | NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI |
| topic | Image and Video Processing Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2505.14064 |