Cover Image

Auditing LLM Benchmarks with Item Response Theory

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Land, Sander, Bikel, Daniel M.
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens
by: Zhao, Zhenyu, et al.
Published: (2026)

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
by: Land, Sander, et al.
Published: (2025)

Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory
by: Zhou, Hongli, et al.
Published: (2025)

Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models
by: Land, Sander, et al.
Published: (2024)

Which Pieces Does Unigram Tokenization Really Need?
by: Land, Sander, et al.
Published: (2025)

Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
by: Cong, Longwei, et al.
Published: (2026)

The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory
by: Schmucker, Robin, et al.
Published: (2025)

Post-training an LLM for RAG? Train on Self-Generated Demonstrations
by: Finlayson, Matthew, et al.
Published: (2025)

RIDE: Difficulty Evolving Perturbation with Item Response Theory for Mathematical Reasoning
by: Li, Xinyuan, et al.
Published: (2025)

JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory
by: Yao, Louie Hong, et al.
Published: (2025)

DECOR: Auditing LLM Deception via Information Manipulation Theory
by: Cai, Linyue, et al.
Published: (2026)

Not What, But How: A Communicative Audit of LLM Response Framing
by: Pawar, Siddhesh Milind, et al.
Published: (2026)

MIRA: A Bilingual Benchmark for Medical Information Response Audit
by: Xu, Mengyu, et al.
Published: (2026)

Concept-Level Explainability for Auditing & Steering LLM Responses
by: Amara, Kenza, et al.
Published: (2025)

Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention
by: Vasudev, Rakshith, et al.
Published: (2026)

SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction
by: Scarlatos, Alexander, et al.
Published: (2025)

Towards Safety and Helpfulness Balanced Responses via Controllable Large Language Models
by: Tuan, Yi-Lin, et al.
Published: (2024)

Enhancing Essay Cohesion Assessment: A Novel Item Response Theory Approach
by: Rosa, Bruno Alexandre, et al.
Published: (2025)

Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks
by: Iyer, Karthik Raghu, et al.
Published: (2026)

Automated Essay Scoring Using Grammatical Variety and Errors with Multi-Task Learning and Item Response Theory
by: Doi, Kosuke, et al.
Published: (2024)

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
by: Tu, Xinming, et al.
Published: (2026)

LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces
by: Kirgis, Peter, et al.
Published: (2026)

Smart Audit System Empowered by LLM
by: Yao, Xu, et al.
Published: (2024)

MAQuA: Adaptive Question-Asking for Multidimensional Mental Health Screening using Item Response Theory
by: Varadarajan, Vasudha, et al.
Published: (2025)

Understanding Likelihood Over-optimisation in Direct Alignment Algorithms
by: Shi, Zhengyan, et al.
Published: (2024)

Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory
by: Uebayashi, Shunki, et al.
Published: (2026)

Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk
by: Chen, Zichen, et al.
Published: (2025)

IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation
by: Lin, Fan, et al.
Published: (2024)

Automated Benchmark Auditing for AI Agents and Large Language Models
by: Wang, Junlin, et al.
Published: (2026)

Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository
by: Kapoor, Radhika, et al.
Published: (2025)

Semi-structured LLM Reasoners Can Be Rigorously Audited
by: Leng, Jixuan, et al.
Published: (2025)

RewardBench 2: Advancing Reward Model Evaluation
by: Malik, Saumya, et al.
Published: (2025)

Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditing
by: Baldini, Ioana, et al.
Published: (2023)

CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era
by: Shi, Kaiwen, et al.
Published: (2026)

UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions
by: Rogoz, Ana-Cristina, et al.
Published: (2024)

Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis
by: Liu, Yunting, et al.
Published: (2024)

Backtracking Improves Generation Safety
by: Zhang, Yiming, et al.
Published: (2024)

Metadata Predictability Is Not Evidence Dependence: An Intervention-Based Audit for Weak-Label Benchmarks
by: Shao, Kan
Published: (2026)

An Audit and Analysis of LLM-Assisted Health Misinformation Jailbreaks Against LLMs
by: Hussain, Ayana, et al.
Published: (2025)

Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks
by: Luo, Zhimeng, et al.
Published: (2025)