WIKIGENBENCH: Exploring Full-length Wikipedia Generation under Real-World Scenario
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916527465299968 |
|---|---|
| author | Zhang, Jiebin Yu, Eugene J. Chen, Qinyu Xiong, Chenhao Zhu, Dawei Qian, Han Song, Mingbo Xiong, Weimin Li, Xiaoguang Liu, Qun Li, Sujian |
| author_facet | Zhang, Jiebin Yu, Eugene J. Chen, Qinyu Xiong, Chenhao Zhu, Dawei Qian, Han Song, Mingbo Xiong, Weimin Li, Xiaoguang Liu, Qun Li, Sujian |
| contents | It presents significant challenges to generate comprehensive and accurate Wikipedia articles for newly emerging events under a real-world scenario. Existing attempts fall short either by focusing only on short snippets or by using metrics that are insufficient to evaluate real-world scenarios. In this paper, we construct WIKIGENBENCH, a new benchmark consisting of 1,320 entries, designed to align with real-world scenarios in both generation and evaluation. For generation, we explore a real-world scenario where structured, full-length Wikipedia articles with citations are generated for new events using input documents from web sources. For evaluation, we integrate systematic metrics and LLM-based metrics to assess the verifiability, organization, and other aspects aligned with real-world scenarios. Based on this benchmark, we conduct extensive experiments using various models within three commonly used frameworks: direct RAG, hierarchical structure-based RAG, and RAG with a fine-tuned generation model. Experimental results show that hierarchical-based methods can generate more comprehensive content, while fine-tuned methods achieve better verifiability. However, even the best methods still show a significant gap compared to existing Wikipedia content, indicating that further research is necessary. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_18264 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WIKIGENBENCH: Exploring Full-length Wikipedia Generation under Real-World Scenario Zhang, Jiebin Yu, Eugene J. Chen, Qinyu Xiong, Chenhao Zhu, Dawei Qian, Han Song, Mingbo Xiong, Weimin Li, Xiaoguang Liu, Qun Li, Sujian Computation and Language It presents significant challenges to generate comprehensive and accurate Wikipedia articles for newly emerging events under a real-world scenario. Existing attempts fall short either by focusing only on short snippets or by using metrics that are insufficient to evaluate real-world scenarios. In this paper, we construct WIKIGENBENCH, a new benchmark consisting of 1,320 entries, designed to align with real-world scenarios in both generation and evaluation. For generation, we explore a real-world scenario where structured, full-length Wikipedia articles with citations are generated for new events using input documents from web sources. For evaluation, we integrate systematic metrics and LLM-based metrics to assess the verifiability, organization, and other aspects aligned with real-world scenarios. Based on this benchmark, we conduct extensive experiments using various models within three commonly used frameworks: direct RAG, hierarchical structure-based RAG, and RAG with a fine-tuned generation model. Experimental results show that hierarchical-based methods can generate more comprehensive content, while fine-tuned methods achieve better verifiability. However, even the best methods still show a significant gap compared to existing Wikipedia content, indicating that further research is necessary. |
| title | WIKIGENBENCH: Exploring Full-length Wikipedia Generation under Real-World Scenario |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.18264 |