Cover Image

Learning to Judge: LLMs Designing and Applying Evaluation Rubrics

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Siro, Clemencia, Aliannejadi, Pourya, Aliannejadi, Mohammad
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs
by: Siro, Clemencia, et al.
Published: (2024)

Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems
by: Siro, Clemencia, et al.
Published: (2024)

Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understanding
by: Ramezan, Kimia, et al.
Published: (2025)

AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs
by: Siro, Clemencia, et al.
Published: (2024)

Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search
by: Yuan, Yifei, et al.
Published: (2024)

Do Images Clarify? A Study on the Effect of Images on Clarifying Questions in Conversational Search
by: Siro, Clemencia, et al.
Published: (2026)

Summarize-Exemplify-Reflect: Data-driven Insight Distillation Empowers LLMs for Few-shot Tabular Classification
by: Yuan, Yifei, et al.
Published: (2025)

Re-Rankers as Relevance Judges
by: Meng, Chuan, et al.
Published: (2026)

Zero-Shot and Efficient Clarification Need Prediction in Conversational Search
by: Lu, Lili, et al.
Published: (2025)

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
by: Hong, Yihan, et al.
Published: (2026)

ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering
by: Lupart, Simon, et al.
Published: (2025)

Ranked List Truncation for Large Language Model-based Re-Ranking
by: Meng, Chuan, et al.
Published: (2024)

Query Performance Prediction using Relevance Judgments Generated by Large Language Models
by: Meng, Chuan, et al.
Published: (2024)

IRLab@iKAT24: Learned Sparse Retrieval with Multi-aspect LLM Query Generation for Conversational Search
by: Lupart, Simon, et al.
Published: (2024)

Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurement
by: Mousavian, Maryam, et al.
Published: (2025)

Generative Retrieval with Few-shot Indexing
by: Askari, Arian, et al.
Published: (2024)

Evaluating the Robustness of Machine Reading Comprehension Models to Low Resource Entity Renaming
by: Siro, Clemencia, et al.
Published: (2023)

DiSCo: LLM Knowledge Distillation for Efficient Sparse Retrieval in Conversational Search
by: Lupart, Simon, et al.
Published: (2024)

On Evaluating LLM Alignment by Evaluating LLMs as Judges
by: Liu, Yixin, et al.
Published: (2025)

ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge
by: Wang, Zhilin, et al.
Published: (2025)

ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
by: Sharma, Manasi, et al.
Published: (2025)

Reinforcement Learning with Rubric Anchors
by: Huang, Zenan, et al.
Published: (2025)

Unlocking Markets: A Multilingual Benchmark to Cross-Market Question Answering
by: Yuan, Yifei, et al.
Published: (2024)

Query Understanding in LLM-based Conversational Information Seeking
by: Yuan, Yifei, et al.
Published: (2025)

RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
by: Li, Gaotang, et al.
Published: (2026)

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
by: Zhou, Yilun, et al.
Published: (2025)

AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
by: Wu, Peilin, et al.
Published: (2026)

Can We Use Large Language Models to Fill Relevance Judgment Holes?
by: Abbasiantaeb, Zahra, et al.
Published: (2024)

Online Rubrics Elicitation from Pairwise Comparisons
by: Rezaei, MohammadHossein, et al.
Published: (2025)

Controlling Gender Bias in Retrieval via a Backpack Architecture
by: Afzali, Amirabbas, et al.
Published: (2025)

Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
by: Gunjal, Anisha, et al.
Published: (2025)

R3: Robust Rubric-Agnostic Reward Models
by: Anugraha, David, et al.
Published: (2025)

Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness
by: Deng, Haotian, et al.
Published: (2025)

Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training
by: Xu, Ran, et al.
Published: (2026)

TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants
by: Aliannejadi, Mohammad, et al.
Published: (2024)

GNN-as-Judge: Unleashing the Power of LLMs for Graph Learning with GNN Feedback
by: Xu, Ruiyao, et al.
Published: (2026)

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
by: Shao, Rulin, et al.
Published: (2025)

Conversational Search: From Fundamentals to Frontiers in the LLM Era
by: Mo, Fengran, et al.
Published: (2025)

JudgeBench: A Benchmark for Evaluating LLM-based Judges
by: Tan, Sijun, et al.
Published: (2024)

Becoming Experienced Judges: Selective Test-Time Learning for Evaluators
by: Jwa, Seungyeon, et al.
Published: (2025)