Cover Image

Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xiong, Lang, Bhargava, Nishant, Hong, Jianhang, Chang, Jeremy, Liu, Haihao, Sharma, Vasu, Zhu, Kevin
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Rewrite-to-Rank: Optimizing Ad Visibility via Retrieval-Aware Text Rewriting
by: Ho, Chloe, et al.
Published: (2025)

Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques
by: Xiong, Lang, et al.
Published: (2025)

TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Models
by: Liu, Joshua, et al.
Published: (2025)

On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
by: Lunardi, Riccardo, et al.
Published: (2025)

ProMoral-Bench: Evaluating Prompting Strategies for Moral Reasoning and Safety in LLMs
by: Thomas, Rohan Subramanian, et al.
Published: (2026)

AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production
by: Kartik, NVJK, et al.
Published: (2025)

Machine Translation Evaluation Benchmark for Wu Chinese: Workflow and Analysis
by: Yu, Hongjian, et al.
Published: (2024)

Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents
by: Choubey, Prafulla Kumar, et al.
Published: (2025)

NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts
by: Gupta, Abhay, et al.
Published: (2025)

xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation
by: Yu, Qingchen, et al.
Published: (2024)

Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores
by: Blackwell, Robert E., et al.
Published: (2024)

Reliable Evaluation and Benchmarks for Statement Autoformalization
by: Poiroux, Auguste, et al.
Published: (2024)

Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoning
by: Baek, Shaun, et al.
Published: (2025)

Probing and Steering Evaluation Awareness of Language Models
by: Nguyen, Jord, et al.
Published: (2025)

Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation
by: AlQadi, Leen, et al.
Published: (2026)

LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing
by: Fein, Daniel, et al.
Published: (2025)

Polarity-Aware Probing for Quantifying Latent Alignment in Language Models
by: Sadiekh, Sabrina, et al.
Published: (2025)

PaperAsk: A Benchmark for Reliability Evaluation of LLMs in Paper Search and Reading
by: Wu, Yutao, et al.
Published: (2025)

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
by: Zhou, Yilun, et al.
Published: (2025)

Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling
by: Maekawa, Seiji, et al.
Published: (2025)

CE-Bench: Towards a Reliable Contrastive Evaluation Benchmark of Interpretability of Sparse Autoencoders
by: Gulko, Alex, et al.
Published: (2025)

Pruning for Performance: Efficient Idiom and Metaphor Classification in Low-Resource Konkani Using mBERT
by: Do, Timothy, et al.
Published: (2025)

Interpreting the Latent Structure of Operator Precedence in Language Models
by: Yugeswardeenoo, Dharunish, et al.
Published: (2025)

MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models
by: Liu, Mianxin, et al.
Published: (2024)

LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Models
by: Liu, Chuang, et al.
Published: (2024)

CLEAR: A Comprehensive Linguistic Evaluation of Argument Rewriting by Large Language Models
by: Huber, Thomas, et al.
Published: (2025)

RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models
by: Shen, Tianhao, et al.
Published: (2023)

CRiskEval: A Chinese Multi-Level Risk Evaluation Benchmark Dataset for Large Language Models
by: Shi, Ling, et al.
Published: (2024)

From Raw Corpora to Domain Benchmarks: Automated Evaluation of LLM Domain Expertise
by: Sharma, Nitin, et al.
Published: (2025)

EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents
by: Dong, Xuan, et al.
Published: (2026)

Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Models
by: Liu, Xinyu, et al.
Published: (2024)

Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments
by: Ye, Bingyang, et al.
Published: (2026)

ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjects
by: Maheshwari, Ayush, et al.
Published: (2025)

Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?
by: Mukherjee, Sourabrata, et al.
Published: (2025)

Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
by: Gu, Zhouhong, et al.
Published: (2023)

FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability
by: Xia, Congying, et al.
Published: (2024)

DCR: Quantifying Data Contamination in LLMs Evaluation
by: Xu, Cheng, et al.
Published: (2025)

Continuous Approximations for Improving Quantization Aware Training of LLMs
by: Li, He, et al.
Published: (2024)

Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibration
by: Begin, James, et al.
Published: (2025)

Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers
by: Liu, Geng, et al.
Published: (2025)