Cover Image

When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	Yu, Fangyi
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The AI Double Standard: Humans Judge All AIs for the Actions of One
by: Manoli, Aikaterina, et al.
Published: (2024)

AIs and Humans with Agency
by: Mumford, David
Published: (2026)

Bootstrapping Developmental AIs: From Simple Competences to Intelligent Human-Compatible AIs
by: Stefik, Mark, et al.
Published: (2023)

Roots and Requirements for Collaborative AIs
by: Stefik, Mark
Published: (2023)

What AIs are not Learning (and Why)
by: Stefik, Mark
Published: (2024)

Do Consumers Accept AIs as Moral Compliance Agents?
by: Nyilasy, Greg, et al.
Published: (2026)

Can Go AIs be adversarially robust?
by: Tseng, Tom, et al.
Published: (2024)

Guillotine: Hypervisors for Isolating Malicious AIs
by: Mickens, James, et al.
Published: (2025)

Draw an Ugly Person An Exploration of Generative AIs Perceptions of Ugliness
by: Kim, Garyoung, et al.
Published: (2025)

How to Count AIs: Individuation and Liability for AI Agents
by: Arbel, Yonathan, et al.
Published: (2026)

Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
by: Thakur, Aman Singh, et al.
Published: (2024)

A mathematical theory of evolution for self-designing AIs
by: Harris, Kenneth D
Published: (2026)

When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback
by: Lang, Leon, et al.
Published: (2024)

Alignment-Process-Outcome: Rethinking How AIs and Humans Collaborate
by: Li, Haichang, et al.
Published: (2026)

Agent-as-a-Judge: Evaluate Agents with Agents
by: Zhuge, Mingchen, et al.
Published: (2024)

Runtime Monitoring and Enforcement of Conditional Fairness in Generative AIs
by: Cheng, Chih-Hong, et al.
Published: (2024)

Embedding Democratic Values into Social Media AIs via Societal Objective Functions
by: Jia, Chenyan, et al.
Published: (2023)

Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
by: Mazeika, Mantas, et al.
Published: (2025)

Agent-as-a-Judge
by: You, Runyang, et al.
Published: (2026)

Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
by: Soumik, Sadman Kabir
Published: (2026)

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries
by: Ravichandran, Sandhanakrishnan, et al.
Published: (2025)

Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
by: Li, Yuran, et al.
Published: (2025)

Bringing AI Participation Down to Scale: A Comment on Open AIs Democratic Inputs to AI Project
by: Moats, David, et al.
Published: (2024)

Cooperative Evolutionary Pressure and Diminishing Returns Might Explain the Fermi Paradox: On What Super-AIs Are Like
by: Vallstrom, Daniel
Published: (2024)

On Evaluating LLM Alignment by Evaluating LLMs as Judges
by: Liu, Yixin, et al.
Published: (2025)

AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
by: Shi, Wentao, et al.
Published: (2026)

Math Education Digital Shadows for facilitating learning with LLMs: Math performance, anxiety and confidence in simulated students and AIs
by: Esposito, Naomi, et al.
Published: (2026)

PentestJudge: Judging Agent Behavior Against Operational Requirements
by: Caldwell, Shane, et al.
Published: (2025)

Murine AI excels at cats and cheese: Structural differences between human and mouse neurons and their implementation in generative AIs
by: Saiga, Rino, et al.
Published: (2024)

Cognitive Architecture Toward Common Ground Sharing Among Humans and Generative AIs: Trial on Model-Model Interactions in Tangram Naming Task
by: Morita, Junya, et al.
Published: (2023)

LLMs Judging LLMs: A Simplex Perspective
by: Vossler, Patrick, et al.
Published: (2025)

Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
by: Gou, Boyu, et al.
Published: (2025)

When AIs Design Themselves: A Triadic Blueprint for the Next Generation
by: GPT-4.5, et al.
Published: (2025)

DPO Learning with LLMs-Judge Signal for Computer Use Agents
by: Luo, Man, et al.
Published: (2025)

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
by: Shi, Lin, et al.
Published: (2024)

MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation
by: Wang, Yutong, et al.
Published: (2025)

JudgeBench: A Benchmark for Evaluating LLM-based Judges
by: Tan, Sijun, et al.
Published: (2024)

EasyJudge: an Easy-to-use Tool for Comprehensive Response Evaluation of LLMs
by: Li, Yijie, et al.
Published: (2024)

ChatGPT and other AIs will change all scientific research: initial reflections on uses and consequences
by: Sampaio, Rafael Cardoso, et al.
Published: (2023)

JAF: Judge Agent Forest
by: Garg, Sahil, et al.
Published: (2026)