Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA
Fuente:
arXiv
Salvato in:
| Autori principali: | , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911258341539840 |
|---|---|
| author | Badshah, Sher Sajjad, Hassan |
| author_facet | Badshah, Sher Sajjad, Hassan |
| contents | The emergence of Large Language Models (LLMs) as chat assistants capable of generating human-like conversations has amplified the need for robust evaluation methods, particularly for open-ended tasks. Conventional metrics such as EM and F1, while useful, are inadequate for capturing the full semantics and contextual depth of such generative outputs. We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs as judges. Through experiments on free-form question-answering tasks, we demonstrate that combining multiple models improves the reliability and accuracy of evaluations, especially in tasks where a single model may struggle. The results indicate a strong correlation with human evaluations, establishing the proposed method as a reliable alternative to traditional metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_09235 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA Badshah, Sher Sajjad, Hassan Computation and Language Artificial Intelligence 68T50, 68T07, 68T20 I.2.0; I.2.7; I.2.2 The emergence of Large Language Models (LLMs) as chat assistants capable of generating human-like conversations has amplified the need for robust evaluation methods, particularly for open-ended tasks. Conventional metrics such as EM and F1, while useful, are inadequate for capturing the full semantics and contextual depth of such generative outputs. We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs as judges. Through experiments on free-form question-answering tasks, we demonstrate that combining multiple models improves the reliability and accuracy of evaluations, especially in tasks where a single model may struggle. The results indicate a strong correlation with human evaluations, establishing the proposed method as a reliable alternative to traditional metrics. |
| title | Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA |
| topic | Computation and Language Artificial Intelligence 68T50, 68T07, 68T20 I.2.0; I.2.7; I.2.2 |
| url | https://arxiv.org/abs/2408.09235 |