Cover Image

Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chern, Steffi, Chern, Ethan, Neubig, Graham, Liu, Pengfei
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Halu-J: Critique-Based Hallucination Judge
by: Wang, Binjie, et al.
Published: (2024)

Combating Adversarial Attacks with Multi-Agent Debate
by: Chern, Steffi, et al.
Published: (2024)

BeHonest: Benchmarking Honesty in Large Language Models
by: Chern, Steffi, et al.
Published: (2024)

Alignment for Honesty
by: Yang, Yuqing, et al.
Published: (2023)

Thinking with Generated Images
by: Chern, Ethan, et al.
Published: (2025)

ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation
by: Chern, Ethan, et al.
Published: (2024)

An Empirical Analysis on Large Language Models in Debate Evaluation
by: Liu, Xinyi, et al.
Published: (2024)

Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme
by: Ma, Yan, et al.
Published: (2025)

LIMO: Less is More for Reasoning
by: Ye, Yixin, et al.
Published: (2025)

Evaluating the Performance of Large Language Models via Debates
by: Moniri, Behrad, et al.
Published: (2024)

On Evaluating LLM Alignment by Evaluating LLMs as Judges
by: Liu, Yixin, et al.
Published: (2025)

Dynamic Evaluation of Large Language Models by Meta Probing Agents
by: Zhu, Kaijie, et al.
Published: (2024)

Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation
by: Zhang, Yichi, et al.
Published: (2025)

FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Models
by: Li, Yiyuan, et al.
Published: (2024)

Effective Strategies for Asynchronous Software Engineering Agents
by: Geng, Jiayi, et al.
Published: (2026)

SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
by: Zhou, Xuhui, et al.
Published: (2023)

Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
by: Bertsch, Amanda, et al.
Published: (2025)

TestAgent: Automatic Benchmarking and Exploratory Interaction for Evaluating LLMs in Vertical Domains
by: Wang, Wanying, et al.
Published: (2024)

MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs
by: Ning, Yucheng, et al.
Published: (2025)

Evaluating Language Models' Evaluations of Games
by: Collins, Katherine M., et al.
Published: (2025)

Evaluating Large Language Models for Abstract Evaluation Tasks: An Empirical Study
by: Liu, Yinuo, et al.
Published: (2026)

Anticipatory Evaluation of Language Models
by: Park, Jungsoo, et al.
Published: (2025)

Evaluating the Meta- and Object-Level Reasoning of Large Language Models for Question Answering
by: Ferguson, Nick, et al.
Published: (2025)

InFoBench: Evaluating Instruction Following Ability in Large Language Models
by: Qin, Yiwei, et al.
Published: (2024)

Evaluating Quantized Large Language Models
by: Li, Shiyao, et al.
Published: (2024)

Can Large Language Model Agents Simulate Human Trust Behavior?
by: Xie, Chengxing, et al.
Published: (2024)

M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation
by: Feng, Zhaopeng, et al.
Published: (2024)

Enabling Scalable Evaluation of Bias Patterns in Medical LLMs
by: Fayyaz, Hamed, et al.
Published: (2024)

O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?
by: Huang, Zhen, et al.
Published: (2024)

Can Language Models Analyze Data? Evaluating Large Language Models for Question Answering over Datasets
by: Xenofontos, Andreas, et al.
Published: (2026)

TrustMH-Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Large Language Models in Mental Health
by: Xiong, Zixin, et al.
Published: (2026)

Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Models
by: Li, Zhuoqun, et al.
Published: (2024)

MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models
by: Li, Jiachun, et al.
Published: (2024)

AgentBench: Evaluating LLMs as Agents
by: Liu, Xiao, et al.
Published: (2023)

RPGBENCH: Evaluating Large Language Models as Role-Playing Game Engines
by: Yu, Pengfei, et al.
Published: (2025)

Reformatted Alignment
by: Fan, Run-Ze, et al.
Published: (2024)

Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems
by: Kazi, Taaha, et al.
Published: (2024)

CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation
by: Wang, Xinchen, et al.
Published: (2025)

League of LLMs: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models
by: Guo, Qianhong, et al.
Published: (2025)

ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation
by: Zheng, Jingnan, et al.
Published: (2024)