Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversight
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915934340382720 |
|---|---|
| author | Ye, Junze Tawfik, Daniel Goodell, Alex J. Kotha, Nikhil V. Buyyounouski, Mark K. Bayati, Mohsen |
| author_facet | Ye, Junze Tawfik, Daniel Goodell, Alex J. Kotha, Nikhil V. Buyyounouski, Mark K. Bayati, Mohsen |
| contents | Reference labels for machine-learning benchmarks are increasingly synthesized with LLM assistance, but their reliability remains underexamined. We audit MedCalc-Bench, a clinical benchmark for medical score computation whose labels were partly derived with LLM assistance, and develop a scalable physician-in-the-loop stewardship pipeline to reassess them. At least 27% of test labels are likely erroneous or incomputable. On a 50-instance subset validated by physicians, our recomputed labels agree with physician ground truth 74% of the time (95% CI, 60-84%) versus 20% for the originals (95% CI, 11-33%). Using original labels to evaluate frontier LLMs underestimates accuracy by 16-23 percentage points. In a controlled reinforcement-learning experiment, a model trained on recomputed labels outperforms one trained on originals by 13.5 percentage points (95% CI, 10.6-16.6%) on physician-labeled instances, and this advantage extends to related medical tasks. LLM-assisted benchmarks can propagate systematic errors into both evaluation and post-training unless actively stewarded. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_19691 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversight Ye, Junze Tawfik, Daniel Goodell, Alex J. Kotha, Nikhil V. Buyyounouski, Mark K. Bayati, Mohsen Artificial Intelligence Applications Reference labels for machine-learning benchmarks are increasingly synthesized with LLM assistance, but their reliability remains underexamined. We audit MedCalc-Bench, a clinical benchmark for medical score computation whose labels were partly derived with LLM assistance, and develop a scalable physician-in-the-loop stewardship pipeline to reassess them. At least 27% of test labels are likely erroneous or incomputable. On a 50-instance subset validated by physicians, our recomputed labels agree with physician ground truth 74% of the time (95% CI, 60-84%) versus 20% for the originals (95% CI, 11-33%). Using original labels to evaluate frontier LLMs underestimates accuracy by 16-23 percentage points. In a controlled reinforcement-learning experiment, a model trained on recomputed labels outperforms one trained on originals by 13.5 percentage points (95% CI, 10.6-16.6%) on physician-labeled instances, and this advantage extends to related medical tasks. LLM-assisted benchmarks can propagate systematic errors into both evaluation and post-training unless actively stewarded. |
| title | Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversight |
| topic | Artificial Intelligence Applications |
| url | https://arxiv.org/abs/2512.19691 |