MARC21: :: Library Catalog

Salvato in:

Dettagli Bibliografici
Autori principali:	Wei, Yishu, Flanders, Adam E., Colak, Errol, Mongan, John, Prevedello, Luciano M, Chen, Po-Hao, Lee, Henrique Min Ho, Szarf, Gilberto, Shoji, Hamilton, Sho, Jason, Andriole, Katherine, Cook, Tessa, Adams, Lisa C., Chu, Linda C., Chung, Maggie, Brusca-Augello, Geraldine, Deva, Djeven P., Singh, Navneet, Tijmes, Felipe Sanchez, Alpert, Jeffrey B., Nguyen, Elsie T., Torigian, Drew A., Hanneman, Kate, Groner, Lauren K, Phan, Alexander, Islam, Ali, Callejas, Matias F., Teles, Gustavo Borges da Silva, Jamal, Faisal, Vazirabad, Maryam, Tejani, Ali, Trivedi, Hari, Kuriki, Paulo, Bhayana, Rajesh, Benishay, Elana T., Lin, Yi, Peng, Yifan, Shih, George
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Computation and Language
Accesso online:	https://arxiv.org/abs/2601.15129
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

_version_	1866915744382451712
author	Wei, Yishu Flanders, Adam E. Colak, Errol Mongan, John Prevedello, Luciano M Chen, Po-Hao Lee, Henrique Min Ho Szarf, Gilberto Shoji, Hamilton Sho, Jason Andriole, Katherine Cook, Tessa Adams, Lisa C. Chu, Linda C. Chung, Maggie Brusca-Augello, Geraldine Deva, Djeven P. Singh, Navneet Tijmes, Felipe Sanchez Alpert, Jeffrey B. Nguyen, Elsie T. Torigian, Drew A. Hanneman, Kate Groner, Lauren K Phan, Alexander Islam, Ali Callejas, Matias F. Teles, Gustavo Borges da Silva Jamal, Faisal Vazirabad, Maryam Tejani, Ali Trivedi, Hari Kuriki, Paulo Bhayana, Rajesh Benishay, Elana T. Lin, Yi Peng, Yifan Shih, George
author_facet	Wei, Yishu Flanders, Adam E. Colak, Errol Mongan, John Prevedello, Luciano M Chen, Po-Hao Lee, Henrique Min Ho Szarf, Gilberto Shoji, Hamilton Sho, Jason Andriole, Katherine Cook, Tessa Adams, Lisa C. Chu, Linda C. Chung, Maggie Brusca-Augello, Geraldine Deva, Djeven P. Singh, Navneet Tijmes, Felipe Sanchez Alpert, Jeffrey B. Nguyen, Elsie T. Torigian, Drew A. Hanneman, Kate Groner, Lauren K Phan, Alexander Islam, Ali Callejas, Matias F. Teles, Gustavo Borges da Silva Jamal, Faisal Vazirabad, Maryam Tejani, Ali Trivedi, Hari Kuriki, Paulo Bhayana, Rajesh Benishay, Elana T. Lin, Yi Peng, Yifan Shih, George
contents	Multimodal large language models have demonstrated comparable performance to that of radiology trainees on multiple-choice board-style exams. However, to develop clinically useful multimodal LLM tools, high-quality benchmarks curated by domain experts are essential. To curate released and holdout datasets of 100 chest radiographic studies each and propose an artificial intelligence (AI)-assisted expert labeling procedure to allow radiologists to label studies more efficiently. A total of 13,735 deidentified chest radiographs and their corresponding reports from the MIDRC were used. GPT-4o extracted abnormal findings from the reports, which were then mapped to 12 benchmark labels with a locally hosted LLM (Phi-4-Reasoning). From these studies, 1,000 were sampled on the basis of the AI-suggested benchmark labels for expert review; the sampling algorithm ensured that the selected studies were clinically relevant and captured a range of difficulty levels. Seventeen chest radiologists participated, and they marked "Agree all", "Agree mostly" or "Disagree" to indicate their assessment of the correctness of the LLM suggested labels. Each chest radiograph was evaluated by three experts. Of these, at least two radiologists selected "Agree All" for 381 radiographs. From this set, 200 were selected, prioritizing those with less common or multiple finding labels, and divided into 100 released radiographs and 100 reserved as the holdout dataset. The holdout dataset is used exclusively by RSNA to independently evaluate different models. A benchmark of 200 chest radiographic studies with 12 benchmark labels was created and made publicly available https://imaging.rsna.org, with each chest radiograph verified by three radiologists. In addition, an AI-assisted labeling procedure was developed to help radiologists label at scale, minimize unnecessary omissions, and support a semicollaborative environment.
format	Preprint
id	arxiv_https___arxiv_org_abs_2601_15129
institution	arXiv
publishDate	2026
record_format	arxiv
spellingShingle	RSNA Large Language Model Benchmark Dataset for Chest Radiographs of Cardiothoracic Disease: Radiologist Evaluation and Validation Enhanced by AI Labels (REVEAL-CXR) Wei, Yishu Flanders, Adam E. Colak, Errol Mongan, John Prevedello, Luciano M Chen, Po-Hao Lee, Henrique Min Ho Szarf, Gilberto Shoji, Hamilton Sho, Jason Andriole, Katherine Cook, Tessa Adams, Lisa C. Chu, Linda C. Chung, Maggie Brusca-Augello, Geraldine Deva, Djeven P. Singh, Navneet Tijmes, Felipe Sanchez Alpert, Jeffrey B. Nguyen, Elsie T. Torigian, Drew A. Hanneman, Kate Groner, Lauren K Phan, Alexander Islam, Ali Callejas, Matias F. Teles, Gustavo Borges da Silva Jamal, Faisal Vazirabad, Maryam Tejani, Ali Trivedi, Hari Kuriki, Paulo Bhayana, Rajesh Benishay, Elana T. Lin, Yi Peng, Yifan Shih, George Computation and Language Multimodal large language models have demonstrated comparable performance to that of radiology trainees on multiple-choice board-style exams. However, to develop clinically useful multimodal LLM tools, high-quality benchmarks curated by domain experts are essential. To curate released and holdout datasets of 100 chest radiographic studies each and propose an artificial intelligence (AI)-assisted expert labeling procedure to allow radiologists to label studies more efficiently. A total of 13,735 deidentified chest radiographs and their corresponding reports from the MIDRC were used. GPT-4o extracted abnormal findings from the reports, which were then mapped to 12 benchmark labels with a locally hosted LLM (Phi-4-Reasoning). From these studies, 1,000 were sampled on the basis of the AI-suggested benchmark labels for expert review; the sampling algorithm ensured that the selected studies were clinically relevant and captured a range of difficulty levels. Seventeen chest radiologists participated, and they marked "Agree all", "Agree mostly" or "Disagree" to indicate their assessment of the correctness of the LLM suggested labels. Each chest radiograph was evaluated by three experts. Of these, at least two radiologists selected "Agree All" for 381 radiographs. From this set, 200 were selected, prioritizing those with less common or multiple finding labels, and divided into 100 released radiographs and 100 reserved as the holdout dataset. The holdout dataset is used exclusively by RSNA to independently evaluate different models. A benchmark of 200 chest radiographic studies with 12 benchmark labels was created and made publicly available https://imaging.rsna.org, with each chest radiograph verified by three radiologists. In addition, an AI-assisted labeling procedure was developed to help radiologists label at scale, minimize unnecessary omissions, and support a semicollaborative environment.
title	RSNA Large Language Model Benchmark Dataset for Chest Radiographs of Cardiothoracic Disease: Radiologist Evaluation and Validation Enhanced by AI Labels (REVEAL-CXR)
topic	Computation and Language
url	https://arxiv.org/abs/2601.15129

if (filemtime($filename) + $maxlifetime < time()) { unlink($filename); $count++; } } return $count; } /** * A function that is called internally when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ protected function saveSession($sessId, $data): bool { $sessFile = $this->path . '/sess_' . $sessId; if ($handle = fopen($sessFile, 'w')) { $return = false; // Lock the file for exclusive access to avoid issues with multiple // processes writing session simultaneously: if (flock($handle, LOCK_EX)) { $return = fwrite($handle, $data); // Make sure that there's no trailing data by truncating the file to // the correct length: ftruncate($handle, strlen($data)); fflush($handle); flock($handle, LOCK_UN); } fclose($handle); if ($return !== false) { return true; } } // If we got this far, something went wrong with the file output... // It is tempting to throw an exception here, but this code is called // outside of the context of exception handling, so all we can do is // echo a message.

// Anecdotal testing Today and Yesterday seems to indicate destroy() // is called by the garbage collector and everything is good. // Something to keep in mind though. return 0; } /** * Write function that is called when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ public function write($sessId, $data): bool { if ($this->writesDisabled) { return true; } return $this->saveSession($sessId, $data); } /** * A function that is called internally when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ abstract protected function saveSession($sessId, $data): bool; }

* * @return void */ public function writeClose() { // The assumption is that we're using PHP's ext/session. // session_write_close() will actually overwrite $_SESSION with an // empty array on completion -- which leads to a mismatch between what // is in the storage object and $_SESSION. To get around this, we // temporarily reset $_SESSION to an array, and then re-link it to // the storage object. // // Additionally, while you _can_ write to $_SESSION following a // session_write_close() operation, no changes made to it will be // flushed to the session handler. As such, we now mark the storage // object isImmutable. $storage = $this->getStorage(); if (! $storage->isImmutable()) { $_SESSION = $storage->toArray(true); session_write_close(); $storage->fromArray($_SESSION); $storage->markImmutable(); } } /** * Attempt to set the session name * * If the session has already been started, or if the name provided fails * validation, an exception will be raised. * * @param string $name * @return SessionManager * @throws Exception\InvalidArgumentException */ public function setName($name) { if ($this->sessionExists()) { throw new Exception\InvalidArgumentException( 'Cannot set session name after a session has already started'

Environment & details:

GET Data empty

POST Data empty

Files empty

Key	Value
language	"it"
VUFIND_SESSION	"f52b8cbf7a377addd0db7967fd0500bb"
ui	"standard"

Session

Key	Value
__Laminas	array:2 [`"_REQUEST_ACCESS_TIME" => 1784972232.9844 "_VALID" => array:1 [ "Laminas\Session\Validator\Id" => "f52b8cbf7a377addd0db7967fd0500bb" ]`]
SessionState	Laminas\Stdlib\ArrayObject {#938}
FlashMessenger	Laminas\Stdlib\ArrayObject {#1578}
SessionHelper	Laminas\Stdlib\ArrayObject {#1775}

Server/Request Data

Key	Value
REDIRECT_VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
REDIRECT_HTTPS	"on"
REDIRECT_SSL_TLS_SNI	"info.umar.mx"
REDIRECT_STATUS	"200"
VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
HTTPS	"on"
SSL_TLS_SNI	"info.umar.mx"
HTTP_ACCEPT	"/"
HTTP_USER_AGENT	"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
HTTP_COOKIE	"language=it; VUFIND_SESSION=f52b8cbf7a377addd0db7967fd0500bb; ui=standard"
HTTP_ACCEPT_ENCODING	"gzip, br, zstd, deflate"
HTTP_HOST	"info.umar.mx"
PATH	"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin"
SERVER_SIGNATURE	"<address>Apache/2.4.66 (Ubuntu) Server at info.umar.mx Port 443</address>\n"
SERVER_SOFTWARE	"Apache/2.4.66 (Ubuntu)"
SERVER_NAME	"info.umar.mx"
SERVER_ADDR	"200.23.222.19"
SERVER_PORT	"443"
REMOTE_ADDR	"216.73.217.169"
DOCUMENT_ROOT	"/usr/local/vufind/public"
REQUEST_SCHEME	"https"
CONTEXT_PREFIX	""
CONTEXT_DOCUMENT_ROOT	"/usr/local/vufind/public"
SERVER_ADMIN	"admin@localhost"
SCRIPT_FILENAME	"/usr/local/vufind/public/index.php"
REMOTE_PORT	"13657"
REDIRECT_URL	""" /Record/\n arxiv_https___arxiv_org_abs_2601_15129/Details """
GATEWAY_INTERFACE	"CGI/1.1"
SERVER_PROTOCOL	"HTTP/1.1"
REQUEST_METHOD	"GET"
QUERY_STRING	""
REQUEST_URI	"/Record/%0Aarxiv_https___arxiv_org_abs_2601_15129/Details"
SCRIPT_NAME	"/index.php"
PHP_SELF	"/index.php"
REQUEST_TIME_FLOAT	1784972232.9767
REQUEST_TIME	1784972232

Environment Variables empty

Registered Handlers

0. Whoops\Handler\PrettyPageHandler

Environment & details:

GET Data empty

POST Data empty

Files empty

Key	Value
language	"it"
VUFIND_SESSION	"f52b8cbf7a377addd0db7967fd0500bb"
ui	"standard"

Session

Key	Value
__Laminas	array:2 [`"_REQUEST_ACCESS_TIME" => 1784972232.9844 "_VALID" => array:1 [ "Laminas\Session\Validator\Id" => "f52b8cbf7a377addd0db7967fd0500bb" ]`]
SessionState	Laminas\Stdlib\ArrayObject {#938}
FlashMessenger	Laminas\Stdlib\ArrayObject {#1578}
SessionHelper	Laminas\Stdlib\ArrayObject {#1775}

Server/Request Data

Key	Value
REDIRECT_VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
REDIRECT_HTTPS	"on"
REDIRECT_SSL_TLS_SNI	"info.umar.mx"
REDIRECT_STATUS	"200"
VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
HTTPS	"on"
SSL_TLS_SNI	"info.umar.mx"
HTTP_ACCEPT	"/"
HTTP_USER_AGENT	"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
HTTP_COOKIE	"language=it; VUFIND_SESSION=f52b8cbf7a377addd0db7967fd0500bb; ui=standard"
HTTP_ACCEPT_ENCODING	"gzip, br, zstd, deflate"
HTTP_HOST	"info.umar.mx"
PATH	"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin"
SERVER_SIGNATURE	"<address>Apache/2.4.66 (Ubuntu) Server at info.umar.mx Port 443</address>\n"
SERVER_SOFTWARE	"Apache/2.4.66 (Ubuntu)"
SERVER_NAME	"info.umar.mx"
SERVER_ADDR	"200.23.222.19"
SERVER_PORT	"443"
REMOTE_ADDR	"216.73.217.169"
DOCUMENT_ROOT	"/usr/local/vufind/public"
REQUEST_SCHEME	"https"
CONTEXT_PREFIX	""
CONTEXT_DOCUMENT_ROOT	"/usr/local/vufind/public"
SERVER_ADMIN	"admin@localhost"
SCRIPT_FILENAME	"/usr/local/vufind/public/index.php"
REMOTE_PORT	"13657"
REDIRECT_URL	""" /Record/\n arxiv_https___arxiv_org_abs_2601_15129/Details """
GATEWAY_INTERFACE	"CGI/1.1"
SERVER_PROTOCOL	"HTTP/1.1"
REQUEST_METHOD	"GET"
QUERY_STRING	""
REQUEST_URI	"/Record/%0Aarxiv_https___arxiv_org_abs_2601_15129/Details"
SCRIPT_NAME	"/index.php"
PHP_SELF	"/index.php"
REQUEST_TIME_FLOAT	1784972232.9767
REQUEST_TIME	1784972232

Environment Variables empty

Registered Handlers

0. Whoops\Handler\PrettyPageHandler

Documenti analoghi

Environment & details:

Environment & details: