Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918115874439168 |
|---|---|
| author | Ashktorab, Zahra Desmond, Michael Pan, Qian Johnson, James M. Cooper, Martin Santillan Daly, Elizabeth M. Nair, Rahul Pedapati, Tejaswini Do, Hyo Jin Geyer, Werner |
| author_facet | Ashktorab, Zahra Desmond, Michael Pan, Qian Johnson, James M. Cooper, Martin Santillan Daly, Elizabeth M. Nair, Rahul Pedapati, Tejaswini Do, Hyo Jin Geyer, Werner |
| contents | Evaluation of large language model (LLM) outputs requires users to make critical judgments about the best outputs across various configurations. This process is costly and takes time given the large amounts of data. LLMs are increasingly used as evaluators to filter training data, evaluate model performance or assist human evaluators with detailed assessments. To support this process, effective front-end tools are critical for evaluation. Two common approaches for using LLMs as evaluators are direct assessment and pairwise comparison. In our study with machine learning practitioners (n=15), each completing 6 tasks yielding 131 evaluations, we explore how task-related factors and assessment strategies influence criteria refinement and user perceptions. Findings show that users performed more evaluations with direct assessment by making criteria task-specific, modifying judgments, and changing the evaluator model. We conclude with recommendations for how systems can better support interactions in LLM-assisted evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_00873 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences Ashktorab, Zahra Desmond, Michael Pan, Qian Johnson, James M. Cooper, Martin Santillan Daly, Elizabeth M. Nair, Rahul Pedapati, Tejaswini Do, Hyo Jin Geyer, Werner Human-Computer Interaction Evaluation of large language model (LLM) outputs requires users to make critical judgments about the best outputs across various configurations. This process is costly and takes time given the large amounts of data. LLMs are increasingly used as evaluators to filter training data, evaluate model performance or assist human evaluators with detailed assessments. To support this process, effective front-end tools are critical for evaluation. Two common approaches for using LLMs as evaluators are direct assessment and pairwise comparison. In our study with machine learning practitioners (n=15), each completing 6 tasks yielding 131 evaluations, we explore how task-related factors and assessment strategies influence criteria refinement and user perceptions. Findings show that users performed more evaluations with direct assessment by making criteria task-specific, modifying judgments, and changing the evaluator model. We conclude with recommendations for how systems can better support interactions in LLM-assisted evaluations. |
| title | Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2410.00873 |