Cover Image

Measuring all the noises of LLM Evals

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	Wang, Sida
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

LLM Probability Concentration: How Alignment Shrinks the Generative Horizon
by: Yang, Chenghao, et al.
Published: (2025)

AcademicEval: Live Long-Context LLM Benchmark
by: Zhang, Haozhen, et al.
Published: (2025)

MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures
by: Ni, Jinjie, et al.
Published: (2024)

LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena
by: Yang, Qingchuan, et al.
Published: (2025)

GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Framework
by: Sansford, Hannah, et al.
Published: (2024)

FedEval-LLM: Federated Evaluation of Large Language Models on Downstream Tasks with Collective Wisdom
by: He, Yuanqin, et al.
Published: (2024)

Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs
by: Wang, Ganghua, et al.
Published: (2025)

QualEval: Qualitative Evaluation for Model Improvement
by: Murahari, Vishvak, et al.
Published: (2023)

HumanEval on Latest GPT Models -- 2024
by: Li, Daniel, et al.
Published: (2024)

ScholarEval: Research Idea Evaluation Grounded in Literature
by: Moussa, Hanane Nour, et al.
Published: (2025)

CHILL at SemEval-2025 Task 2: You Can't Just Throw Entities and Hope -- Make Your LLM to Get Them Right
by: Lee, Jaebok, et al.
Published: (2025)

HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation
by: Li, Mingxuan, et al.
Published: (2025)

AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
by: Andriushchenko, Maksym, et al.
Published: (2024)

RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation
by: Bondarenko, Ivan, et al.
Published: (2026)

Measuring the Depth of LLM Unlearning via Activation Patching
by: Lee, Jaeung, et al.
Published: (2026)

Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation
by: Young, Richard J.
Published: (2026)

Measuring and Reducing LLM Hallucination without Gold-Standard Answers
by: Wei, Jiaheng, et al.
Published: (2024)

Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks
by: Gong, Linyuan, et al.
Published: (2024)

Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
by: Dubois, Yann, et al.
Published: (2024)

100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances
by: Pacchiardi, Lorenzo, et al.
Published: (2024)

It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertainty
by: Guo, Kevin H., et al.
Published: (2026)

KFinEval-Pilot: A Comprehensive Benchmark Suite for Korean Financial Language Understanding
by: Hwang, Bokwang, et al.
Published: (2025)

StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation
by: Cao, Boxi, et al.
Published: (2024)

DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models
by: Chung, Tsz Ting, et al.
Published: (2025)

LyS at SemEval 2025 Task 8: Zero-Shot Code Generation for Tabular QA
by: Gude, Adrián, et al.
Published: (2025)

IITK at SemEval-2024 Task 4: Hierarchical Embeddings for Detection of Persuasion Techniques in Memes
by: Chikoti, Shreenaga, et al.
Published: (2024)

SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense
by: Jiang, Yifan, et al.
Published: (2024)

CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures
by: Pandey, Punya Syon, et al.
Published: (2025)

AIMA at SemEval-2024 Task 3: Simple Yet Powerful Emotion Cause Pair Analysis
by: Kure, Alireza Ghahramani, et al.
Published: (2025)

FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
by: Ming, Yifei, et al.
Published: (2024)

IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages?
by: Aravapalli, Akhilesh, et al.
Published: (2024)

ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks
by: Yu, Xiaodong, et al.
Published: (2023)

IITK at SemEval-2024 Task 1: Contrastive Learning and Autoencoders for Semantic Textual Relatedness in Multilingual Texts
by: Basak, Udvas, et al.
Published: (2024)

X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects
by: Liu, Minqian, et al.
Published: (2023)

MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors
by: Macina, Jakub, et al.
Published: (2025)

AutoEval Done Right: Using Synthetic Data for Model Evaluation
by: Boyeau, Pierre, et al.
Published: (2024)

Large Language Models aren't all that you need
by: Holla, Kiran Voderhobli, et al.
Published: (2024)

AIMA at SemEval-2024 Task 10: History-Based Emotion Recognition in Hindi-English Code-Mixed Conversations
by: Abootorabi, Mohammad Mahdi, et al.
Published: (2025)

UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without machine translation
by: Dipta, Shubhashis Roy, et al.
Published: (2024)

TLDR at SemEval-2024 Task 2: T5-generated clinical-Language summaries for DeBERTa Report Analysis
by: Das, Spandan, et al.
Published: (2024)