SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917398533111808 |
|---|---|
| author | Huang, Xiyang Lin, Jiawei Wu, Keying Huang, Jiaxin Yang, Kailai Wei, Renxiong zeng, Cheng Xiang, Jiayi Kuang, Ziyan Peng, Min Xie, Qianqian Ananiadou, Sophia |
| author_facet | Huang, Xiyang Lin, Jiawei Wu, Keying Huang, Jiaxin Yang, Kailai Wei, Renxiong zeng, Cheng Xiang, Jiayi Kuang, Ziyan Peng, Min Xie, Qianqian Ananiadou, Sophia |
| contents | Current video benchmarks for multimodal large language models (MLLMs) focus on event recognition, temporal ordering, and long-context recall, but overlook a harder capability required for expert procedural judgment: tracking how ongoing interactions update the procedural state and thereby determine the correctness of later actions. We introduce SiMing-Bench, the first benchmark for evaluating this capability from full-length clinical skill videos. It targets rubric-grounded process-level judgment of whether interaction-driven state updates preserve procedural correctness across an entire workflow. SiMing-Bench is instantiated with SiMing-Score, a physician-annotated dataset of real clinical skill examination videos spanning cardiopulmonary resuscitation, automated external defibrillator operation, and bag-mask ventilation, each paired with a standardized step-wise rubric and dual-expert labels. Across diverse open- and closed-source MLLMs, we observe consistently weak agreement with physician judgments. Moreover, weak performance on rubric-defined intermediate steps persists even when overall procedure-level correlation appears acceptable, suggesting that coarse global assessment substantially overestimates current models' procedural judgment ability. Additional analyses with binary step judgment and step-aligned clips indicate that the bottleneck is not merely fine-grained scoring or temporal localization, but modeling how continuous interactions update procedural state over time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_09037 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos Huang, Xiyang Lin, Jiawei Wu, Keying Huang, Jiaxin Yang, Kailai Wei, Renxiong zeng, Cheng Xiang, Jiayi Kuang, Ziyan Peng, Min Xie, Qianqian Ananiadou, Sophia Computer Vision and Pattern Recognition Computation and Language Human-Computer Interaction Current video benchmarks for multimodal large language models (MLLMs) focus on event recognition, temporal ordering, and long-context recall, but overlook a harder capability required for expert procedural judgment: tracking how ongoing interactions update the procedural state and thereby determine the correctness of later actions. We introduce SiMing-Bench, the first benchmark for evaluating this capability from full-length clinical skill videos. It targets rubric-grounded process-level judgment of whether interaction-driven state updates preserve procedural correctness across an entire workflow. SiMing-Bench is instantiated with SiMing-Score, a physician-annotated dataset of real clinical skill examination videos spanning cardiopulmonary resuscitation, automated external defibrillator operation, and bag-mask ventilation, each paired with a standardized step-wise rubric and dual-expert labels. Across diverse open- and closed-source MLLMs, we observe consistently weak agreement with physician judgments. Moreover, weak performance on rubric-defined intermediate steps persists even when overall procedure-level correlation appears acceptable, suggesting that coarse global assessment substantially overestimates current models' procedural judgment ability. Additional analyses with binary step judgment and step-aligned clips indicate that the bottleneck is not merely fine-grained scoring or temporal localization, but modeling how continuous interactions update procedural state over time. |
| title | SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos |
| topic | Computer Vision and Pattern Recognition Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2604.09037 |