How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918048013746176 |
|---|---|
| author | Zeng, Qiming Yan, Xiao Luo, Hao Lin, Yuhao Wang, Yuxiang Fu, Fangcheng Du, Bo Xu, Quanqing Jiang, Jiawei |
| author_facet | Zeng, Qiming Yan, Xiao Luo, Hao Lin, Yuhao Wang, Yuxiang Fu, Fangcheng Du, Bo Xu, Quanqing Jiang, Jiawei |
| contents | By retrieving contexts from knowledge graphs, graph-based retrieval-augmented generation (GraphRAG) enhances large language models (LLMs) to generate quality answers for user questions. Many GraphRAG methods have been proposed and reported inspiring performance in answer quality. However, we observe that the current answer evaluation framework for GraphRAG has two critical flaws, i.e., unrelated questions and evaluation biases, which may lead to biased or even wrong conclusions on performance. To tackle the two flaws, we propose an unbiased evaluation framework that uses graph-text-grounded question generation to produce questions that are more related to the underlying dataset and an unbiased evaluation procedure to eliminate the biases in LLM-based answer assessment. We apply our unbiased framework to evaluate 3 representative GraphRAG methods and find that their performance gains are much more moderate than reported previously. Although our evaluation framework may still have flaws, it calls for scientific evaluations to lay solid foundations for GraphRAG research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06331 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG Zeng, Qiming Yan, Xiao Luo, Hao Lin, Yuhao Wang, Yuxiang Fu, Fangcheng Du, Bo Xu, Quanqing Jiang, Jiawei Computation and Language Artificial Intelligence Information Retrieval By retrieving contexts from knowledge graphs, graph-based retrieval-augmented generation (GraphRAG) enhances large language models (LLMs) to generate quality answers for user questions. Many GraphRAG methods have been proposed and reported inspiring performance in answer quality. However, we observe that the current answer evaluation framework for GraphRAG has two critical flaws, i.e., unrelated questions and evaluation biases, which may lead to biased or even wrong conclusions on performance. To tackle the two flaws, we propose an unbiased evaluation framework that uses graph-text-grounded question generation to produce questions that are more related to the underlying dataset and an unbiased evaluation procedure to eliminate the biases in LLM-based answer assessment. We apply our unbiased framework to evaluate 3 representative GraphRAG methods and find that their performance gains are much more moderate than reported previously. Although our evaluation framework may still have flaws, it calls for scientific evaluations to lay solid foundations for GraphRAG research. |
| title | How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG |
| topic | Computation and Language Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2506.06331 |