Quantitative LLM Judges
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911227305787392 |
|---|---|
| author | Sahoo, Aishwarya Karnuthala, Jeevana Kruthi Budhwani, Tushar Parmanand Agarwal, Pranchal Vaidyanathan, Sankaran Siu, Alexa Dernoncourt, Franck Healey, Jennifer Lipka, Nedim Rossi, Ryan Bhattacharya, Uttaran Kveton, Branislav |
| author_facet | Sahoo, Aishwarya Karnuthala, Jeevana Kruthi Budhwani, Tushar Parmanand Agarwal, Pranchal Vaidyanathan, Sankaran Siu, Alexa Dernoncourt, Franck Healey, Jennifer Lipka, Nedim Rossi, Ryan Bhattacharya, Uttaran Kveton, Branislav |
| contents | LLM-as-a-judge is a framework where a large language model (LLM) evaluates the output of another LLM. While LLMs excel at producing qualitative textual evaluations, they often struggle to predict human preferences and numeric scores. We propose quantitative LLM judges, which align evaluation scores of existing LLM judges to humans in a given domain using regression models. The models are trained to improve the score of the original judge using its rationale and score. We present four quantitative judges for different types of absolute and relative feedback, which showcases the generality and versatility of our framework. Our framework is more computationally efficient than supervised fine-tuning and can be more statistically efficient when human feedback is limited, which is expected in practice. We validate these claims empirically on four datasets using two base judges. Our experiments show that quantitative judges can improve the predictive power of existing judges through post-hoc modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02945 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Quantitative LLM Judges Sahoo, Aishwarya Karnuthala, Jeevana Kruthi Budhwani, Tushar Parmanand Agarwal, Pranchal Vaidyanathan, Sankaran Siu, Alexa Dernoncourt, Franck Healey, Jennifer Lipka, Nedim Rossi, Ryan Bhattacharya, Uttaran Kveton, Branislav Computation and Language Machine Learning LLM-as-a-judge is a framework where a large language model (LLM) evaluates the output of another LLM. While LLMs excel at producing qualitative textual evaluations, they often struggle to predict human preferences and numeric scores. We propose quantitative LLM judges, which align evaluation scores of existing LLM judges to humans in a given domain using regression models. The models are trained to improve the score of the original judge using its rationale and score. We present four quantitative judges for different types of absolute and relative feedback, which showcases the generality and versatility of our framework. Our framework is more computationally efficient than supervised fine-tuning and can be more statistically efficient when human feedback is limited, which is expected in practice. We validate these claims empirically on four datasets using two base judges. Our experiments show that quantitative judges can improve the predictive power of existing judges through post-hoc modeling. |
| title | Quantitative LLM Judges |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.02945 |