Does UMBRELA Work on Other LLMs?
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913939145621504 |
|---|---|
| author | Farzi, Naghmeh Dietz, Laura |
| author_facet | Farzi, Naghmeh Dietz, Laura |
| contents | We reproduce the UMBRELA LLM Judge evaluation framework across a range of large language models (LLMs) to assess its generalizability beyond the original study. Our investigation evaluates how LLM choice affects relevance assessment accuracy, focusing on leaderboard rank correlation and per-label agreement metrics. Results demonstrate that UMBRELA with DeepSeek V3 obtains very comparable performance to GPT-4o (used in original work). For LLaMA-3.3-70B we obtain slightly lower performance, which further degrades with smaller LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_09483 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Does UMBRELA Work on Other LLMs? Farzi, Naghmeh Dietz, Laura Information Retrieval H.3.3; I.2.7 We reproduce the UMBRELA LLM Judge evaluation framework across a range of large language models (LLMs) to assess its generalizability beyond the original study. Our investigation evaluates how LLM choice affects relevance assessment accuracy, focusing on leaderboard rank correlation and per-label agreement metrics. Results demonstrate that UMBRELA with DeepSeek V3 obtains very comparable performance to GPT-4o (used in original work). For LLaMA-3.3-70B we obtain slightly lower performance, which further degrades with smaller LLMs. |
| title | Does UMBRELA Work on Other LLMs? |
| topic | Information Retrieval H.3.3; I.2.7 |
| url | https://arxiv.org/abs/2507.09483 |