SurveyEval: Towards Comprehensive Evaluation of LLM-Generated Academic Surveys
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912743364231168 |
|---|---|
| author | Zhao, Jiahao Zhang, Shuaixing Xu, Nan Wang, Lei |
| author_facet | Zhao, Jiahao Zhang, Shuaixing Xu, Nan Wang, Lei |
| contents | LLM-based automatic survey systems are transforming how users acquire information from the web by integrating retrieval, organization, and content synthesis into end-to-end generation pipelines. While recent works focus on developing new generation pipelines, how to evaluate such complex systems remains a significant challenge. To this end, we introduce SurveyEval, a comprehensive benchmark that evaluates automatically generated surveys across three dimensions: overall quality, outline coherence, and reference accuracy. We extend the evaluation across 7 subjects and augment the LLM-as-a-Judge framework with human references to strengthen evaluation-human alignment. Evaluation results show that while general long-text or paper-writing systems tend to produce lower-quality surveys, specialized survey-generation systems are able to deliver substantially higher-quality results. We envision SurveyEval as a scalable testbed to understand and improve automatic survey systems across diverse subjects and evaluation criteria. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_02763 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SurveyEval: Towards Comprehensive Evaluation of LLM-Generated Academic Surveys Zhao, Jiahao Zhang, Shuaixing Xu, Nan Wang, Lei Computation and Language Artificial Intelligence LLM-based automatic survey systems are transforming how users acquire information from the web by integrating retrieval, organization, and content synthesis into end-to-end generation pipelines. While recent works focus on developing new generation pipelines, how to evaluate such complex systems remains a significant challenge. To this end, we introduce SurveyEval, a comprehensive benchmark that evaluates automatically generated surveys across three dimensions: overall quality, outline coherence, and reference accuracy. We extend the evaluation across 7 subjects and augment the LLM-as-a-Judge framework with human references to strengthen evaluation-human alignment. Evaluation results show that while general long-text or paper-writing systems tend to produce lower-quality surveys, specialized survey-generation systems are able to deliver substantially higher-quality results. We envision SurveyEval as a scalable testbed to understand and improve automatic survey systems across diverse subjects and evaluation criteria. |
| title | SurveyEval: Towards Comprehensive Evaluation of LLM-Generated Academic Surveys |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2512.02763 |