Cover Image

Medical Large Language Model Benchmarks Should Prioritize Construct Validity

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Alaa, Ahmed, Hartvigsen, Thomas, Golchini, Niloufar, Dutta, Shiladitya, Dean, Frances, Raji, Inioluwa Deborah, Zack, Travis
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Multi-lingual Functional Evaluation for Large Language Models
by: Ojewale, Victor, et al.
Published: (2025)

ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room
by: Mehandru, Nikita, et al.
Published: (2025)

Model Editing with Graph-Based External Memory
by: Atri, Yash Kumar, et al.
Published: (2025)

Measuring what Matters: Construct Validity in Large Language Model Benchmarks
by: Bean, Andrew M., et al.
Published: (2025)

Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes
by: Christ, Bryan R., et al.
Published: (2024)

Concrete Problems in AI Safety, Revisited
by: Raji, Inioluwa Deborah, et al.
Published: (2023)

Evaluating Prediction-based Interventions with Human Decision Makers In Mind
by: Raji, Inioluwa Deborah, et al.
Published: (2025)

"I Searched for a Religious Song in Amharic and Got Sexual Content Instead": Investigating Online Harm in Low-Resourced Languages on YouTube
by: Nigatu, Hellina Hailu, et al.
Published: (2024)

TAXI: Evaluating Categorical Knowledge Editing for Language Models
by: Powell, Derek, et al.
Published: (2024)

Continually Self-Improving Language Models for Bariatric Surgery Question--Answering
by: Atri, Yash Kumar, et al.
Published: (2025)

Lifelong Knowledge Editing requires Better Regularization
by: Gupta, Akshat, et al.
Published: (2025)

PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Models
by: Jain, Devansh, et al.
Published: (2024)

Can Language Models Identify Side Effects of Breast Cancer Radiation Treatments?
by: Seah, Natalie, et al.
Published: (2026)

Norm Growth and Stability Challenges in Localized Sequential Knowledge Editing
by: Gupta, Akshat, et al.
Published: (2025)

MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
by: Chen, Shan, et al.
Published: (2025)

Large Language Model Benchmarks in Medical Tasks
by: Yan, Lawrence K. Q., et al.
Published: (2024)

Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
by: Gallifant, Jack, et al.
Published: (2024)

AI Safety Should Prioritize the Future of Work
by: Hazra, Sanchaita, et al.
Published: (2025)

From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms
by: Dai, Jessica, et al.
Published: (2025)

The Data Addition Dilemma
by: Shen, Judy Hanwen, et al.
Published: (2024)

User Profile with Large Language Models: Construction, Updating, and Benchmarking
by: Prottasha, Nusrat Jahan, et al.
Published: (2025)

CEQuest: Benchmarking Large Language Models for Construction Estimation
by: Wu, Yanzhao, et al.
Published: (2025)

PeruMedQA: Benchmarking Large Language Models (LLMs) on Peruvian Medical Exams -- Dataset Construction and Evaluation
by: Carrillo-Larco, Rodrigo M., et al.
Published: (2025)

Evaluating Temporal Consistency in Multi-Turn Language Models
by: Atri, Yash Kumar, et al.
Published: (2026)

Arabic Large Language Models for Medical Text Generation
by: Allam, Abdulrahman, et al.
Published: (2025)

Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs
by: Song, Xiangchen, et al.
Published: (2025)

CORAL: Expert-Curated medical Oncology Reports to Advance Language Model Inference
by: Sushil, Madhumita, et al.
Published: (2023)

Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions
by: Chen, Hanjie, et al.
Published: (2024)

Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks
by: Vishwanath, Krithik, et al.
Published: (2025)

Construction of a Japanese Financial Benchmark for Large Language Models
by: Hirano, Masanori
Published: (2024)

Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization
by: Zhang, Zhexin, et al.
Published: (2023)

ReasonEdit: Editing Vision-Language Models using Human Reasoning
by: Qiu, Jiaxing, et al.
Published: (2026)

Ontology Matching with Large Language Models and Prioritized Depth-First Search
by: Taboada, Maria, et al.
Published: (2025)

Aggregated Individual Reporting for Post-Deployment Evaluation
by: Dai, Jessica, et al.
Published: (2025)

Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation
by: Ren, Zhiyao, et al.
Published: (2026)

A Systematic Analysis of Declining Medical Safety Messaging in Generative AI Models
by: Sharma, Sonali, et al.
Published: (2025)

Identifying Reasons for Contraceptive Switching from Real-World Data Using Large Language Models
by: Miao, Brenda Y., et al.
Published: (2024)

Scaling Laws for Discriminative Classification in Large Language Models
by: Wyatte, Dean, et al.
Published: (2024)

Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning
by: Kim, Jonathan, et al.
Published: (2025)

MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering
by: Alonso, Iñigo, et al.
Published: (2024)