Copertina

Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Blackwell, Robert E., Barry, Jon, Cohn, Anthony G.
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Evaluating the Ability of Large Language Models to Reason about Cardinal Directions
di: Cohn, Anthony G, et al.
Pubblicazione: (2024)

Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited
di: Cohn, Anthony G, et al.
Pubblicazione: (2025)

Can Large Language Models Reason about the Region Connection Calculus?
di: Cohn, Anthony G, et al.
Pubblicazione: (2024)

QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculi
di: Cohn, Anthony G., et al.
Pubblicazione: (2026)

Exploring Spatial Representations in the Historical Lake District Texts with LLM-based Relation Extraction
di: Haris, Erum, et al.
Pubblicazione: (2024)

Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis
di: Cohn, Clayton, et al.
Pubblicazione: (2024)

Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning
di: Li, Fangjun, et al.
Pubblicazione: (2024)

Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark
di: Li, Fangjun, et al.
Pubblicazione: (2024)

To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation
di: Cheng, Xiang, et al.
Pubblicazione: (2024)

An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoning
di: Kaur, Navdeep, et al.
Pubblicazione: (2025)

Evaluating Scoring Bias in LLM-as-a-Judge
di: Li, Qingquan, et al.
Pubblicazione: (2025)

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
di: Thellmann, Klaudia-Doris, et al.
Pubblicazione: (2026)

Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
di: Cao, Yixin, et al.
Pubblicazione: (2025)

The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements
di: Zhao, Bingchen, et al.
Pubblicazione: (2025)

Benchmark^2: Systematic Evaluation of LLM Benchmarks
di: Qian, Qi, et al.
Pubblicazione: (2026)

PredictaBoard: Benchmarking LLM Score Predictability
di: Pacchiardi, Lorenzo, et al.
Pubblicazione: (2025)

Ranking Over Scoring: Towards Reliable and Robust Automated Evaluation of LLM-Generated Medical Explanatory Arguments
di: De la Iglesia, Iker, et al.
Pubblicazione: (2024)

Tools in the Loop: Quantifying Uncertainty of LLM Question Answering Systems That Use Tools
di: Lymperopoulos, Panagiotis, et al.
Pubblicazione: (2025)

NC-Bench: An LLM Benchmark for Evaluating Conversational Competence
di: Moore, Robert J., et al.
Pubblicazione: (2026)

TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness
di: Zheng, Danna, et al.
Pubblicazione: (2024)

The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring
di: Cacioli, Jon-Paul
Pubblicazione: (2026)

Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM
di: Lim, Zheng Wei, et al.
Pubblicazione: (2024)

Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation
di: Lee, Dongryeol, et al.
Pubblicazione: (2024)

From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
di: Laskar, Md Tahmid Rahman, et al.
Pubblicazione: (2026)

Quantifying the Persona Effect in LLM Simulations
di: Hu, Tiancheng, et al.
Pubblicazione: (2024)

IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation
di: Tran, Khanh-Tung, et al.
Pubblicazione: (2025)

Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation
di: Huang, Sukai, et al.
Pubblicazione: (2024)

Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench
di: Perlitz, Yotam, et al.
Pubblicazione: (2024)

Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling
di: Maekawa, Seiji, et al.
Pubblicazione: (2025)

In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scores
di: Tang, Zeyu, et al.
Pubblicazione: (2026)

Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals
di: Cacioli, Jon-Paul
Pubblicazione: (2026)

Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction
di: Sheng, Huanxin, et al.
Pubblicazione: (2025)

HalluLens: LLM Hallucination Benchmark
di: Bang, Yejin, et al.
Pubblicazione: (2025)

Every Response Counts: Quantifying Uncertainty of LLM-based Multi-Agent Systems through Tensor Decomposition
di: Chen, Tiejin, et al.
Pubblicazione: (2026)

CoTAL: Human-in-the-Loop Prompt Engineering for Generalizable Formative Assessment Scoring
di: Cohn, Clayton, et al.
Pubblicazione: (2025)

A Reproducibility Study of LLM-Based Query Reformulation
di: Bigdeli, Amin, et al.
Pubblicazione: (2026)

Foundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustness
di: Giordano, Luca, et al.
Pubblicazione: (2025)

Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
di: Atasoy, I. F., et al.
Pubblicazione: (2026)

Evaluating the Relevance of Uncertainty Estimators for LLM Hallucination
di: Agnimo, Yedidia, et al.
Pubblicazione: (2026)

Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation
di: Chen, Junjie, et al.
Pubblicazione: (2026)