Cover Image

FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Lee, Gyubok, Bach, Elea, Yang, Eric, Pollard, Tom, Johnson, Alistair, Choi, Edward, jia, Yugang, Lee, Jong Ha
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

SCARE: A Benchmark for SQL Correction and Question Answerability Classification for Reliable EHR Question Answering
by: Lee, Gyubok, et al.
Published: (2025)

AgentBench: Evaluating LLMs as Agents
by: Liu, Xiao, et al.
Published: (2023)

H-AdminSim: A Multi-Agent Simulator for Realistic Hospital Administrative Workflows with FHIR Integration
by: Lee, Jun-Min, et al.
Published: (2026)

From Conversation to Query Execution: Benchmarking User and Tool Interactions for EHR Database Agents
by: Lee, Gyubok, et al.
Published: (2025)

EHR-SeqSQL : A Sequential Text-to-SQL Dataset For Interactively Exploring Electronic Health Records
by: Ryu, Jaehee, et al.
Published: (2024)

FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering
by: Choi, Chanyeol, et al.
Published: (2025)

EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records
by: Kwon, Yeonsu, et al.
Published: (2024)

MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents
by: Jiang, Yixing, et al.
Published: (2025)

MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools
by: Guo, Zikang, et al.
Published: (2025)

AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Data
by: Rashidian, Sina, et al.
Published: (2025)

Reinforcement Learning for Tool-Calling Agents in Fast Healthcare Interoperability Resources (FHIR)
by: Knorr, Marius S., et al.
Published: (2026)

TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring
by: Lee, Gyubok, et al.
Published: (2024)

Overview of the EHRSQL 2024 Shared Task on Reliable Text-to-SQL Modeling on Electronic Health Records
by: Lee, Gyubok, et al.
Published: (2024)

FHIRconnect: Towards a seamless integration of openEHR and FHIR
by: Kohler, Severin, et al.
Published: (2025)

Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL
by: Yang, Yongjin, et al.
Published: (2024)

FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records
by: Frew, Michael, et al.
Published: (2026)

The Geometry of Queries: Query-Based Innovations in Retrieval-Augmented Generation for Healthcare QA
by: Yang, Eric, et al.
Published: (2024)

STEER: Inference-Time Risk Control via Constrained Quality-Diversity Search
by: Yang, Eric, et al.
Published: (2026)

EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records
by: Lee, Gyubok, et al.
Published: (2023)

Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions
by: Yun, Taedong, et al.
Published: (2025)

Spezi Data Pipeline: Streamlining FHIR-based Interoperable Digital Health Data Workflows
by: Bikia, Vasiliki, et al.
Published: (2025)

Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
by: Garg, Spandan, et al.
Published: (2025)

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
by: Merrill, Mike A., et al.
Published: (2026)

WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting
by: Styles, Olly, et al.
Published: (2024)

PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
by: Liu, Ruoqi, et al.
Published: (2026)

ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents
by: Jeon, YoungHoon, et al.
Published: (2026)

CXReasonAgent: Evidence-Grounded Diagnostic Reasoning Agent for Chest X-rays
by: Lee, Hyungyung, et al.
Published: (2026)

EHRSummarizer: A Privacy-Aware, FHIR-Native Reference Architecture for Source-Grounded EHR Summarization
by: Kazemzadeh, Houman, et al.
Published: (2026)

ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents
by: Lee, Seunghyun, et al.
Published: (2026)

KoBBQ: Korean Bias Benchmark for Question Answering
by: Jin, Jiho, et al.
Published: (2023)

DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards
by: Kartha, Aaryaman, et al.
Published: (2025)

Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models
by: Park, Jean, et al.
Published: (2024)

KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs
by: Kim, Haechan, et al.
Published: (2026)

An Entity Linking Agent for Question Answering
by: Luo, Yajie, et al.
Published: (2025)

CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays
by: Lee, Hyungyung, et al.
Published: (2025)

UTSA-NLP at ArchEHR-QA 2025: Improving EHR Question Answering via Self-Consistency Prompting
by: Shields-Menard, Sara, et al.
Published: (2025)

Jamendo-MT-QA: A Benchmark for Multi-Track Comparative Music Question Answering
by: Koh, Junyoung, et al.
Published: (2026)

AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering
by: Kuan, Chun-Yi, et al.
Published: (2026)

TableBench: A Comprehensive and Complex Benchmark for Table Question Answering
by: Wu, Xianjie, et al.
Published: (2024)

BenchQC: A Benchmarking Toolkit for Quantum Computation
by: Pollard, Nia, et al.
Published: (2025)