LLMJudge: LLMs for Relevance Judgments
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911992809259008 |
|---|---|
| author | Rahmani, Hossein A. Yilmaz, Emine Craswell, Nick Mitra, Bhaskar Thomas, Paul Clarke, Charles L. A. Aliannejadi, Mohammad Siro, Clemencia Faggioli, Guglielmo |
| author_facet | Rahmani, Hossein A. Yilmaz, Emine Craswell, Nick Mitra, Bhaskar Thomas, Paul Clarke, Charles L. A. Aliannejadi, Mohammad Siro, Clemencia Faggioli, Guglielmo |
| contents | The LLMJudge challenge is organized as part of the LLM4Eval workshop at SIGIR 2024. Test collections are essential for evaluating information retrieval (IR) systems. The evaluation and tuning of a search system is largely based on relevance labels, which indicate whether a document is useful for a specific search and user. However, collecting relevance judgments on a large scale is costly and resource-intensive. Consequently, typical experiments rely on third-party labelers who may not always produce accurate annotations. The LLMJudge challenge aims to explore an alternative approach by using LLMs to generate relevance judgments. Recent studies have shown that LLMs can generate reliable relevance judgments for search systems. However, it remains unclear which LLMs can match the accuracy of human labelers, which prompts are most effective, how fine-tuned open-source LLMs compare to closed-source LLMs like GPT-4, whether there are biases in synthetically generated data, and if data leakage affects the quality of generated labels. This challenge will investigate these questions, and the collected data will be released as a package to support automatic relevance judgment research in information retrieval and search. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_08896 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLMJudge: LLMs for Relevance Judgments Rahmani, Hossein A. Yilmaz, Emine Craswell, Nick Mitra, Bhaskar Thomas, Paul Clarke, Charles L. A. Aliannejadi, Mohammad Siro, Clemencia Faggioli, Guglielmo Information Retrieval The LLMJudge challenge is organized as part of the LLM4Eval workshop at SIGIR 2024. Test collections are essential for evaluating information retrieval (IR) systems. The evaluation and tuning of a search system is largely based on relevance labels, which indicate whether a document is useful for a specific search and user. However, collecting relevance judgments on a large scale is costly and resource-intensive. Consequently, typical experiments rely on third-party labelers who may not always produce accurate annotations. The LLMJudge challenge aims to explore an alternative approach by using LLMs to generate relevance judgments. Recent studies have shown that LLMs can generate reliable relevance judgments for search systems. However, it remains unclear which LLMs can match the accuracy of human labelers, which prompts are most effective, how fine-tuned open-source LLMs compare to closed-source LLMs like GPT-4, whether there are biases in synthetically generated data, and if data leakage affects the quality of generated labels. This challenge will investigate these questions, and the collected data will be released as a package to support automatic relevance judgment research in information retrieval and search. |
| title | LLMJudge: LLMs for Relevance Judgments |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2408.08896 |