GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910758395183104 |
|---|---|
| author | Deshpande, Darshan Ravi, Selvan Sunitha CH-Wang, Sky Mielczarek, Bartosz Kannappan, Anand Qian, Rebecca |
| author_facet | Deshpande, Darshan Ravi, Selvan Sunitha CH-Wang, Sky Mielczarek, Bartosz Kannappan, Anand Qian, Rebecca |
| contents | The LLM-as-judge paradigm is increasingly being adopted for automated evaluation of model outputs. While LLM judges have shown promise on constrained evaluation tasks, closed source LLMs display critical shortcomings when deployed in real world applications due to challenges of fine grained metrics and explainability, while task specific evaluation models lack cross-domain generalization. We introduce GLIDER, a powerful 3B evaluator LLM that can score any text input and associated context on arbitrary user defined criteria. GLIDER shows higher Pearson's correlation than GPT-4o on FLASK and greatly outperforms prior evaluation models, achieving comparable performance to LLMs 17x its size. GLIDER supports fine-grained scoring, multilingual reasoning, span highlighting and was trained on 685 domains and 183 criteria. Extensive qualitative analysis shows that GLIDER scores are highly correlated with human judgments, with 91.3% human agreement. We have open-sourced GLIDER to facilitate future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14140 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking Deshpande, Darshan Ravi, Selvan Sunitha CH-Wang, Sky Mielczarek, Bartosz Kannappan, Anand Qian, Rebecca Computation and Language Artificial Intelligence The LLM-as-judge paradigm is increasingly being adopted for automated evaluation of model outputs. While LLM judges have shown promise on constrained evaluation tasks, closed source LLMs display critical shortcomings when deployed in real world applications due to challenges of fine grained metrics and explainability, while task specific evaluation models lack cross-domain generalization. We introduce GLIDER, a powerful 3B evaluator LLM that can score any text input and associated context on arbitrary user defined criteria. GLIDER shows higher Pearson's correlation than GPT-4o on FLASK and greatly outperforms prior evaluation models, achieving comparable performance to LLMs 17x its size. GLIDER supports fine-grained scoring, multilingual reasoning, span highlighting and was trained on 685 domains and 183 criteria. Extensive qualitative analysis shows that GLIDER scores are highly correlated with human judgments, with 91.3% human agreement. We have open-sourced GLIDER to facilitate future research. |
| title | GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.14140 |