Cover Image

Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hu, Renjun, Cheng, Yi, Meng, Libin, Xia, Jiaxin, Zong, Yi, Shi, Xing, Lin, Wei
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Investigating Non-Transitivity in LLM-as-a-Judge
by: Xu, Yi, et al.
Published: (2025)

Large Language Models Could Be Rote Learners
by: Xu, Yuyang, et al.
Published: (2025)

Training Domain Draft Models for Speculative Decoding: Best Practices and Insights
by: Hong, Fenglu, et al.
Published: (2025)

Arithmetic Feature Interaction Is Necessary for Deep Tabular Learning
by: Cheng, Yi, et al.
Published: (2024)

Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
by: Liu, Yixin, et al.
Published: (2026)

Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
by: Li, Zhuochun, et al.
Published: (2026)

SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression
by: Xu, Yuyang, et al.
Published: (2025)

Quantifying and Mitigating Self-Preference Bias of LLM Judges
by: Yang, Jinming, et al.
Published: (2026)

JudgeBench: A Benchmark for Evaluating LLM-based Judges
by: Tan, Sijun, et al.
Published: (2024)

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
by: Shi, Lin, et al.
Published: (2024)

JTCSE: Joint Tensor-Modulus Constraints and Cross-Attention for Unsupervised Contrastive Learning of Sentence Embeddings
by: Zong, Tianyu, et al.
Published: (2025)

On Designing Effective RL Reward at Training Time for LLM Reasoning
by: Gao, Jiaxuan, et al.
Published: (2024)

On Evaluating LLM Alignment by Evaluating LLMs as Judges
by: Liu, Yixin, et al.
Published: (2025)

Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings
by: Xu, Austin, et al.
Published: (2025)

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
by: Hong, Yihan, et al.
Published: (2026)

Training-free LLM Merging for Multi-task Learning
by: Fu, Zichuan, et al.
Published: (2025)

Preparing Lessons for Progressive Training on Language Models
by: Pan, Yu, et al.
Published: (2024)

The Lessons of Developing Process Reward Models in Mathematical Reasoning
by: Zhang, Zhenru, et al.
Published: (2025)

Enhancing Hepatopathy Clinical Trial Efficiency: A Secure, Large Language Model-Powered Pre-Screening Pipeline
by: Gui, Xiongbin, et al.
Published: (2025)

CultureLLM: Incorporating Cultural Differences into Large Language Models
by: Li, Cheng, et al.
Published: (2024)

I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
by: Hu, Xing, et al.
Published: (2024)

JuStRank: Benchmarking LLM Judges for System Ranking
by: Gera, Ariel, et al.
Published: (2024)

Tuning LLM Judge Design Decisions for 1/1000 of the Cost
by: Salinas, David, et al.
Published: (2025)

ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
by: You, Haoran, et al.
Published: (2024)

LLM-AutoDP: Automatic Data Processing via LLM Agents for Model Fine-tuning
by: Huang, Wei, et al.
Published: (2026)

Schema Lineage Extraction at Scale: Multilingual Pipelines, Composite Evaluation, and Language-Model Benchmarks
by: Yin, Jiaqi, et al.
Published: (2025)

Towards Best Practices for Open Datasets for LLM Training
by: Baack, Stefan, et al.
Published: (2025)

Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders
by: Jing, Yi, et al.
Published: (2026)

CLEAR: Error Analysis via LLM-as-a-Judge Made Easy
by: Yehudai, Asaf, et al.
Published: (2025)

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
by: Gupta, Manan, et al.
Published: (2026)

Learning Dynamics of LLM Finetuning
by: Ren, Yi, et al.
Published: (2024)

BAGEN: Are LLM Agents Budget-Aware?
by: Lin, Yuxiang, et al.
Published: (2026)

Ask Again, Then Fail: Large Language Models' Vacillations in Judgment
by: Xie, Qiming, et al.
Published: (2023)

Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers
by: Bouchard, Dylan, et al.
Published: (2025)

J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
by: Whitehouse, Chenxi, et al.
Published: (2025)

When LLM Judge Scores Look Good but Best-of-N Decisions Fail
by: Landesberg, Eddie
Published: (2026)

Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
by: Xu, Ran, et al.
Published: (2025)

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge
by: Zhang, Wenbo, et al.
Published: (2026)

Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation
by: Alam, Firoj, et al.
Published: (2026)

Open or Closed LLM for Lesser-Resourced Languages? Lessons from Greek
by: Pavlopoulos, John, et al.
Published: (2025)