No-Human in the Loop: Agentic Evaluation at Scale for Recommendation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909887450054656 |
|---|---|
| author | Zhang, Tao Yao, Kehui Ma, Luyi Chen, Jiao Maragheh, Reza Yousefi Zhao, Kai Xu, Jianpeng Korpeoglu, Evren Kumar, Sushant Achan, Kannan |
| author_facet | Zhang, Tao Yao, Kehui Ma, Luyi Chen, Jiao Maragheh, Reza Yousefi Zhao, Kai Xu, Jianpeng Korpeoglu, Evren Kumar, Sushant Achan, Kannan |
| contents | Evaluating large language models (LLMs) as judges is increasingly critical for building scalable and trustworthy evaluation pipelines. We present ScalingEval, a large-scale benchmarking study that systematically compares 36 LLMs, including GPT, Gemini, Claude, and Llama, across multiple product categories using a consensus-driven evaluation protocol. Our multi-agent framework aggregates pattern audits and issue codes into ground-truth labels via scalable majority voting, enabling reproducible comparison of LLM evaluators without human annotation. Applied to large-scale complementary-item recommendation, the benchmark reports four key findings: (i) Anthropic Claude 3.5 Sonnet achieves the highest decision confidence; (ii) Gemini 1.5 Pro offers the best overall performance across categories; (iii) GPT-4o provides the most favorable latency-accuracy-cost tradeoff; and (iv) GPT-OSS 20B leads among open-source models. Category-level analysis shows strong consensus in structured domains (Electronics, Sports) but persistent disagreement in lifestyle categories (Clothing, Food). These results establish ScalingEval as a reproducible benchmark and evaluation protocol for LLMs as judges, with actionable guidance on scaling, reliability, and model family tradeoffs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_03051 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | No-Human in the Loop: Agentic Evaluation at Scale for Recommendation Zhang, Tao Yao, Kehui Ma, Luyi Chen, Jiao Maragheh, Reza Yousefi Zhao, Kai Xu, Jianpeng Korpeoglu, Evren Kumar, Sushant Achan, Kannan Artificial Intelligence Information Retrieval Evaluating large language models (LLMs) as judges is increasingly critical for building scalable and trustworthy evaluation pipelines. We present ScalingEval, a large-scale benchmarking study that systematically compares 36 LLMs, including GPT, Gemini, Claude, and Llama, across multiple product categories using a consensus-driven evaluation protocol. Our multi-agent framework aggregates pattern audits and issue codes into ground-truth labels via scalable majority voting, enabling reproducible comparison of LLM evaluators without human annotation. Applied to large-scale complementary-item recommendation, the benchmark reports four key findings: (i) Anthropic Claude 3.5 Sonnet achieves the highest decision confidence; (ii) Gemini 1.5 Pro offers the best overall performance across categories; (iii) GPT-4o provides the most favorable latency-accuracy-cost tradeoff; and (iv) GPT-OSS 20B leads among open-source models. Category-level analysis shows strong consensus in structured domains (Electronics, Sports) but persistent disagreement in lifestyle categories (Clothing, Food). These results establish ScalingEval as a reproducible benchmark and evaluation protocol for LLMs as judges, with actionable guidance on scaling, reliability, and model family tradeoffs. |
| title | No-Human in the Loop: Agentic Evaluation at Scale for Recommendation |
| topic | Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2511.03051 |