Copertina

Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Wei, Hui, He, Shenghua, Xia, Tian, Liu, Fei, Wong, Andy, Lin, Jingyang, Han, Mei
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning
di: Lin, Jingyang, et al.
Pubblicazione: (2025)

PlanGenLLMs: A Modern Survey of LLM Planning Capabilities
di: Wei, Hui, et al.
Pubblicazione: (2025)

An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks
di: Zhou, Xin, et al.
Pubblicazione: (2025)

Evaluating Metrics for Safety with LLM-as-Judges
di: Clegg, Kester, et al.
Pubblicazione: (2025)

SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
di: Mou, Yutao, et al.
Pubblicazione: (2024)

On Evaluating LLM Alignment by Evaluating LLMs as Judges
di: Liu, Yixin, et al.
Pubblicazione: (2025)

Agri-CPJ: A Training-Free Explainable Framework for Agricultural Pest Diagnosis Using Caption-Prompt-Judge and LLM-as-a-Judge
di: Zhang, Wentao, et al.
Pubblicazione: (2026)

CPJ: Explainable Agricultural Pest Diagnosis via Caption-Prompt-Judge with LLM-Judged Refinement
di: Zhang, Wentao, et al.
Pubblicazione: (2025)

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
di: Shi, Lin, et al.
Pubblicazione: (2024)

Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
di: He, Shenghua, et al.
Pubblicazione: (2025)

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
di: Eigler, Lukáš, et al.
Pubblicazione: (2026)

LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data
di: Meisenbacher, Stephen, et al.
Pubblicazione: (2025)

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
di: Rao, Delip, et al.
Pubblicazione: (2026)

HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
di: Yang, Langqi, et al.
Pubblicazione: (2025)

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
di: Bellibatlu, Rohith Reddy, et al.
Pubblicazione: (2026)

Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge
di: Wu, Junjie, et al.
Pubblicazione: (2026)

An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4
di: Huang, Hui, et al.
Pubblicazione: (2024)

Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation
di: Alam, Firoj, et al.
Pubblicazione: (2026)

Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
di: Verga, Pat, et al.
Pubblicazione: (2024)

PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics
di: Larionov, Daniil, et al.
Pubblicazione: (2024)

ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents
di: Chang, Hwan, et al.
Pubblicazione: (2025)

Systematically Analyzing Prompt Injection Vulnerabilities in Diverse LLM Architectures
di: Benjamin, Victoria, et al.
Pubblicazione: (2024)

Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models
di: Le, Hieu Xuan, et al.
Pubblicazione: (2026)

BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge
di: Tong, Terry, et al.
Pubblicazione: (2025)

Teach2Eval: An Indirect Evaluation Method for LLM by Judging How It Teaches
di: Zhou, Yuhang, et al.
Pubblicazione: (2025)

When Metrics Disagree: Automatic Similarity vs. LLM-as-a-Judge for Clinical Dialogue Evaluation
di: Sun, Bian, et al.
Pubblicazione: (2026)

CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation
di: Zhu, Ziyi, et al.
Pubblicazione: (2026)

CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
di: Jiang, Hongchao, et al.
Pubblicazione: (2025)

Evaluating Scoring Bias in LLM-as-a-Judge
di: Li, Qingquan, et al.
Pubblicazione: (2025)

Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation
di: Lin, Wei-Hsiang, et al.
Pubblicazione: (2025)

Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction
di: Sheng, Huanxin, et al.
Pubblicazione: (2025)

Think-J: Learning to Think for Generative LLM-as-a-Judge
di: Huang, Hui, et al.
Pubblicazione: (2025)

Transferable Modeling Strategies for Low-Resource LLM Tasks: A Prompt and Alignment-Based Approach
di: Lyu, Shuangquan, et al.
Pubblicazione: (2025)

Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
di: Wu, Tianhao, et al.
Pubblicazione: (2024)

Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks
di: Maloyan, Narek, et al.
Pubblicazione: (2025)

Rethinking Atomic Decomposition for LLM Judges: A Prompt-Controlled Study of Reference-Grounded QA Evaluation
di: Zhang, Xinran
Pubblicazione: (2026)

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
di: Zhou, Yilun, et al.
Pubblicazione: (2025)

Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA
di: Belmadani, Ikram, et al.
Pubblicazione: (2026)

Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judge
di: Cai, Yunna, et al.
Pubblicazione: (2025)

CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Tasks
di: Lin, Jiayu, et al.
Pubblicazione: (2026)