EvalAssist: A Human-Centered Tool for LLM-as-a-Judge
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908604725985280 |
|---|---|
| author | Ashktorab, Zahra Geyer, Werner Desmond, Michael Daly, Elizabeth M. Cooper, Martin Santillan Pan, Qian Miehling, Erik Pedapati, Tejaswini Do, Hyo Jin |
| author_facet | Ashktorab, Zahra Geyer, Werner Desmond, Michael Daly, Elizabeth M. Cooper, Martin Santillan Pan, Qian Miehling, Erik Pedapati, Tejaswini Do, Hyo Jin |
| contents | With the broad availability of large language models and their ability to generate vast outputs using varied prompts and configurations, determining the best output for a given task requires an intensive evaluation process, one where machine learning practitioners must decide how to assess the outputs and then carefully carry out the evaluation. This process is both time-consuming and costly. As practitioners work with an increasing number of models, they must now evaluate outputs to determine which model and prompt performs best for a given task. LLMs are increasingly used as evaluators to filter training data, evaluate model performance, assess harms and risks, or assist human evaluators with detailed assessments. We present EvalAssist, a framework that simplifies the LLM-as-a-judge workflow. The system provides an online criteria development environment, where users can interactively build, test, and share custom evaluation criteria in a structured and portable format. We support a set of LLM-based evaluation pipelines that leverage off-the-shelf LLMs and use a prompt-chaining approach we developed and contributed to the UNITXT open-source library. Additionally, our system also includes specially trained evaluators to detect harms and risks in LLM outputs. We have deployed the system internally in our organization with several hundreds of users. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_02186 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EvalAssist: A Human-Centered Tool for LLM-as-a-Judge Ashktorab, Zahra Geyer, Werner Desmond, Michael Daly, Elizabeth M. Cooper, Martin Santillan Pan, Qian Miehling, Erik Pedapati, Tejaswini Do, Hyo Jin Human-Computer Interaction With the broad availability of large language models and their ability to generate vast outputs using varied prompts and configurations, determining the best output for a given task requires an intensive evaluation process, one where machine learning practitioners must decide how to assess the outputs and then carefully carry out the evaluation. This process is both time-consuming and costly. As practitioners work with an increasing number of models, they must now evaluate outputs to determine which model and prompt performs best for a given task. LLMs are increasingly used as evaluators to filter training data, evaluate model performance, assess harms and risks, or assist human evaluators with detailed assessments. We present EvalAssist, a framework that simplifies the LLM-as-a-judge workflow. The system provides an online criteria development environment, where users can interactively build, test, and share custom evaluation criteria in a structured and portable format. We support a set of LLM-based evaluation pipelines that leverage off-the-shelf LLMs and use a prompt-chaining approach we developed and contributed to the UNITXT open-source library. Additionally, our system also includes specially trained evaluators to detect harms and risks in LLM outputs. We have deployed the system internally in our organization with several hundreds of users. |
| title | EvalAssist: A Human-Centered Tool for LLM-as-a-Judge |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2507.02186 |