JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911376486694912 |
|---|---|
| author | Shi, Zhichao Jiang, Xuhui Xu, Chengjin Yao, Cangli Ma, Shengjia Shen, Yinghan Li, Zixuan Guo, Jian Wang, Yuanzhuo |
| author_facet | Shi, Zhichao Jiang, Xuhui Xu, Chengjin Yao, Cangli Ma, Shengjia Shen, Yinghan Li, Zixuan Guo, Jian Wang, Yuanzhuo |
| contents | Current evaluation methods for large language models (LLMs) primarily rely on static benchmarks, presenting two major challenges: limited knowledge coverage and fixed difficulties that mismatch with the evaluated LLMs. These limitations lead to superficial assessments of LLM knowledge, thereby impeding the targeted model optimizations. To bridge this gap, we propose JudgeAgent, a knowledge-driven and dynamic evaluation framework for LLMs. To address the challenge of limited knowledge coverage, JudgeAgent leverages LLM agents equipped with context graphs to traverse knowledge structures systematically for question generation. Furthermore, to mitigate data contamination and difficulty mismatch, it adopts a difficulty-adaptive and multi-turn interview mechanism. Thereby, JudgeAgent can achieve comprehensive evaluations and facilitate more effective improvement of LLMs. Empirical results demonstrate that JudgeAgent enables more comprehensive evaluations and facilitates effective model iterations, highlighting the potential of this knowledge-driven and dynamic evaluation paradigm. The source code is available on https://github.com/DataArcTech/JudgeAgent. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_02097 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation Shi, Zhichao Jiang, Xuhui Xu, Chengjin Yao, Cangli Ma, Shengjia Shen, Yinghan Li, Zixuan Guo, Jian Wang, Yuanzhuo Computation and Language Artificial Intelligence Current evaluation methods for large language models (LLMs) primarily rely on static benchmarks, presenting two major challenges: limited knowledge coverage and fixed difficulties that mismatch with the evaluated LLMs. These limitations lead to superficial assessments of LLM knowledge, thereby impeding the targeted model optimizations. To bridge this gap, we propose JudgeAgent, a knowledge-driven and dynamic evaluation framework for LLMs. To address the challenge of limited knowledge coverage, JudgeAgent leverages LLM agents equipped with context graphs to traverse knowledge structures systematically for question generation. Furthermore, to mitigate data contamination and difficulty mismatch, it adopts a difficulty-adaptive and multi-turn interview mechanism. Thereby, JudgeAgent can achieve comprehensive evaluations and facilitate more effective improvement of LLMs. Empirical results demonstrate that JudgeAgent enables more comprehensive evaluations and facilitates effective model iterations, highlighting the potential of this knowledge-driven and dynamic evaluation paradigm. The source code is available on https://github.com/DataArcTech/JudgeAgent. |
| title | JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.02097 |